Post on 11-Nov-2014
description
BETA work in progress I add every training new material and tune current
material
Sogeti Netherlands
Microsoft Test and
Lab
Manager Training
Guide A description and short reference guide covering Microsoft Test and Lab
Manager training course ldquoTesting Practices with ALM toolsrdquo
Clemens Reijnen -- VERSION 04 882011
The training course This guide is an extract from the two and three day course provided by me It spans the
complete testing lifecycle and the tool usages It will look at the infrastructure
implications and testing practices in formal and in agile teams But the main focus stays
on the usages of the Microsoft Visual Studio testing tools the knowledge you need to get
starting with it the practices you must have to work with it in real live and how you can
bend the tools with extensibility and normal use to your team needs
The course and this guide is work in progress It is not a testing training (I expect you
already have testing knowledge) if you need that test process information I refer to the
TMap website from Sogeti where you can find tons of information This training guide
follows the TMap testing lifecycle
Contents The training course 1
Application Lifecycle Management 4
Test Lifecycle - Planning Phase 8
Work items 9
Lab 1 Create team projects 10
Lab 2 Create user stories and tasks for the employee application 20
Work Item Queries 21
Reports Basic 24
Microsoft Test Manager 28
Test Lifecycle ndash The Control Phase I 29
Test Plans 29
Test Configurations 32
Test Suites 32
Lab 3 Create and configure test plan 34
TCMEXE 36
Test Lifecycle ndash Specification Phase 37
Test Cases 38
Shared Steps 41
Test Data Iterations 42
Lab 4 Create test cases 43
Test Lifecycle ndash Execution Phase 46
Test Run 47
Test Runner 48
Lab 5 Execute the created test cases 50
Test Case Execution Results 51
Test Lifecycle ndash Control Phase II 56
File a Bug 57
Verify a bug 60
Lab 6 file a bug 61
Bug Reports 67
Test Impact 67
Lab 7 test impact 69
Create test from bug 70
Lab 8 Create an exploratory bug and create a test case from it 71
Test Lifecycle - Infrastructure Phase 72
On Physical Environments with Microsoft Test Manager 74
Lab 9 Install and configure the test controller and test agent 79
Lab 10 Setup Flavor D 79
Test Lifecycle - Completion Phase 80
Test automation with Visual Studio 2010 81
When is test automation the investment worth 83
The investment levels explained 84
Lab 13 Create CodedUI 88
The UIMap 89
Data driven CodedUI tests 90
Lab 14 Execute the CodedUI during the build from Visual Studio 91
Setting up Flavor C 91
Lab 16 Execute the codedUI from MTM 93
Setting up Flavor D 93
Lab 17 Execute the codedUI during the build from MTM 95
Setting up Flavor E 95
On Virtualized Environments with Lab Management and SCVMM 96
Extensibility 99
Tools 99
Custome Data Adapter 100
Azure and test automation 101
Test Manager and an agile process 101
Intelitrace 101
Appendix 106
proposed training agenda 2 day training 106
Student and trainer Environment 107
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
From Forrester
httpiicomcomcnwk1dhtmlitpSerena_Life_Cycle_Managementpdf
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
The training course This guide is an extract from the two and three day course provided by me It spans the
complete testing lifecycle and the tool usages It will look at the infrastructure
implications and testing practices in formal and in agile teams But the main focus stays
on the usages of the Microsoft Visual Studio testing tools the knowledge you need to get
starting with it the practices you must have to work with it in real live and how you can
bend the tools with extensibility and normal use to your team needs
The course and this guide is work in progress It is not a testing training (I expect you
already have testing knowledge) if you need that test process information I refer to the
TMap website from Sogeti where you can find tons of information This training guide
follows the TMap testing lifecycle
Contents The training course 1
Application Lifecycle Management 4
Test Lifecycle - Planning Phase 8
Work items 9
Lab 1 Create team projects 10
Lab 2 Create user stories and tasks for the employee application 20
Work Item Queries 21
Reports Basic 24
Microsoft Test Manager 28
Test Lifecycle ndash The Control Phase I 29
Test Plans 29
Test Configurations 32
Test Suites 32
Lab 3 Create and configure test plan 34
TCMEXE 36
Test Lifecycle ndash Specification Phase 37
Test Cases 38
Shared Steps 41
Test Data Iterations 42
Lab 4 Create test cases 43
Test Lifecycle ndash Execution Phase 46
Test Run 47
Test Runner 48
Lab 5 Execute the created test cases 50
Test Case Execution Results 51
Test Lifecycle ndash Control Phase II 56
File a Bug 57
Verify a bug 60
Lab 6 file a bug 61
Bug Reports 67
Test Impact 67
Lab 7 test impact 69
Create test from bug 70
Lab 8 Create an exploratory bug and create a test case from it 71
Test Lifecycle - Infrastructure Phase 72
On Physical Environments with Microsoft Test Manager 74
Lab 9 Install and configure the test controller and test agent 79
Lab 10 Setup Flavor D 79
Test Lifecycle - Completion Phase 80
Test automation with Visual Studio 2010 81
When is test automation the investment worth 83
The investment levels explained 84
Lab 13 Create CodedUI 88
The UIMap 89
Data driven CodedUI tests 90
Lab 14 Execute the CodedUI during the build from Visual Studio 91
Setting up Flavor C 91
Lab 16 Execute the codedUI from MTM 93
Setting up Flavor D 93
Lab 17 Execute the codedUI during the build from MTM 95
Setting up Flavor E 95
On Virtualized Environments with Lab Management and SCVMM 96
Extensibility 99
Tools 99
Custome Data Adapter 100
Azure and test automation 101
Test Manager and an agile process 101
Intelitrace 101
Appendix 106
proposed training agenda 2 day training 106
Student and trainer Environment 107
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
From Forrester
httpiicomcomcnwk1dhtmlitpSerena_Life_Cycle_Managementpdf
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Contents The training course 1
Application Lifecycle Management 4
Test Lifecycle - Planning Phase 8
Work items 9
Lab 1 Create team projects 10
Lab 2 Create user stories and tasks for the employee application 20
Work Item Queries 21
Reports Basic 24
Microsoft Test Manager 28
Test Lifecycle ndash The Control Phase I 29
Test Plans 29
Test Configurations 32
Test Suites 32
Lab 3 Create and configure test plan 34
TCMEXE 36
Test Lifecycle ndash Specification Phase 37
Test Cases 38
Shared Steps 41
Test Data Iterations 42
Lab 4 Create test cases 43
Test Lifecycle ndash Execution Phase 46
Test Run 47
Test Runner 48
Lab 5 Execute the created test cases 50
Test Case Execution Results 51
Test Lifecycle ndash Control Phase II 56
File a Bug 57
Verify a bug 60
Lab 6 file a bug 61
Bug Reports 67
Test Impact 67
Lab 7 test impact 69
Create test from bug 70
Lab 8 Create an exploratory bug and create a test case from it 71
Test Lifecycle - Infrastructure Phase 72
On Physical Environments with Microsoft Test Manager 74
Lab 9 Install and configure the test controller and test agent 79
Lab 10 Setup Flavor D 79
Test Lifecycle - Completion Phase 80
Test automation with Visual Studio 2010 81
When is test automation the investment worth 83
The investment levels explained 84
Lab 13 Create CodedUI 88
The UIMap 89
Data driven CodedUI tests 90
Lab 14 Execute the CodedUI during the build from Visual Studio 91
Setting up Flavor C 91
Lab 16 Execute the codedUI from MTM 93
Setting up Flavor D 93
Lab 17 Execute the codedUI during the build from MTM 95
Setting up Flavor E 95
On Virtualized Environments with Lab Management and SCVMM 96
Extensibility 99
Tools 99
Custome Data Adapter 100
Azure and test automation 101
Test Manager and an agile process 101
Intelitrace 101
Appendix 106
proposed training agenda 2 day training 106
Student and trainer Environment 107
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
From Forrester
httpiicomcomcnwk1dhtmlitpSerena_Life_Cycle_Managementpdf
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Create test from bug 70
Lab 8 Create an exploratory bug and create a test case from it 71
Test Lifecycle - Infrastructure Phase 72
On Physical Environments with Microsoft Test Manager 74
Lab 9 Install and configure the test controller and test agent 79
Lab 10 Setup Flavor D 79
Test Lifecycle - Completion Phase 80
Test automation with Visual Studio 2010 81
When is test automation the investment worth 83
The investment levels explained 84
Lab 13 Create CodedUI 88
The UIMap 89
Data driven CodedUI tests 90
Lab 14 Execute the CodedUI during the build from Visual Studio 91
Setting up Flavor C 91
Lab 16 Execute the codedUI from MTM 93
Setting up Flavor D 93
Lab 17 Execute the codedUI during the build from MTM 95
Setting up Flavor E 95
On Virtualized Environments with Lab Management and SCVMM 96
Extensibility 99
Tools 99
Custome Data Adapter 100
Azure and test automation 101
Test Manager and an agile process 101
Intelitrace 101
Appendix 106
proposed training agenda 2 day training 106
Student and trainer Environment 107
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
From Forrester
httpiicomcomcnwk1dhtmlitpSerena_Life_Cycle_Managementpdf
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Application Lifecycle
Management
You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and donrsquot see and use its full value
IT organizations spend billions of dollars a year on development life-cycle tools that donrsquot
play well together The result For most shops application life-cycle management (ALM)
mdash the coordination of development life-cycle activities mdash is still largely a manual process
Todayrsquos ALM suites donrsquot offer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations But tomorrowrsquos ALM platforms will do much
better by providing common services to practitioner tools These solutions will be easier
to implement maintain and employ And at the end of the day theyrsquoll enable
development organizations to build better software
From Forrester
httpiicomcomcnwk1dhtmlitpSerena_Life_Cycle_Managementpdf
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
(tool vendors ALM proposition)
httpwww-01ibmcomsoftwarerationalalm
httpwwwmicrosoftcomvisualstudioen-ussolutionsmanagement
httpsh10078www1hpcomcdahpmsdisplaymainhpms_contentjspzn=btoampcp=1-
11^37618_4000_100__
Software development and all roles involved are supposed to strive for one common
goal providing added value to the customer It does not matter what kind of application
needs to be implemented All tasks and roles need to reach for this one single goal The
synergy between roles processes and tools is also commonly known as Application
Lifecycle Management [ALM] Notions like accountability governance and compliance
are regularly used when talking about ALM But all these notions refer to the simple
notion of cooperation
There would be a better mutual understanding if developers and testers would have more
insight in each otherrsquos work and challenges Annoyances disappear and people work
together on a common cause Visual Studio Team System 2010 will better support this
cooperation By giving the tester aids for performing hisher tasks and making himher a
first-class-citizen within the Team Foundation Server heshe has and gives more insight in tasks to be done progress on these tasks and what steps still need to be performed
Aside from this insight which often means quite a change of culture the tester has
better insight in and access to the ALM-Artifacts from other roles For example use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition Cooperation not only occurs by giving insight in work activities but also by
working on common products
httpwwwclemensreijnennlpost20090421Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Editionaspx
httpwwwclemensreijnennlpost20080218ALM-Definitionsaspx
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
When talking about Application Lifecycle Management [ALM] terms like accountability
governance and compliance are used All of them refer back to ldquoworking togetherrdquo how
do we work together during the application lifecycle ALM is not about tools itrsquos about
working together Working together seamless and flexible while staying in control being
measurable and responsible All the roles in the application lifecycle have a part in this
collaboration effort Tools can help but isnrsquotrsquo the core driver
There are lots of examples of wrong interpreted business requirements miss
communication between development en test [Rob describes a very real example]
applications which wonrsquot run in production operations who donrsquot understand the
applications All of them resulting in more work more faults more costs or even only
costs and no application because the project was unplugged Most of these projects faults these extra costs are a slip in communication
Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management Most organizational parts have already some
kind of process methodology in place Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success factor for lowering the costs and raise the business value of IT investments
Tools can help with this goal Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management But without a
plan how people should collaborate and communicate tools are useless
httpwwwclemensreijnennlpost20090213Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycleaspx
Application Lifecycle Management and a Collaborative Culture
Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect More specifically we need to address some of the worries
and obstacles people encounter when collaborating using technology
The three most important concerns are
Trust Trust is a condition for social interaction People will only work with people
companies tools and information they know they can trust Before we can expect
collaboration to take off online there must be a way for people to get this ldquotrustrdquo And a
topic closely associated with trust when it refers to people is Identity
Collaborative culture If one individual is the greatest collaborator in the world he or
she is probably not getting anywhere Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached A
collaborative culture consists of many things including
Collaborative leadership
Shared goals
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Shared model of the truth and
Rules or norms
Reward Changing the way people work takes effort so it must be clear for the parties
involved what they will gain at a personal level from collaborating in a new way
Surprisingly a ldquorewardrdquo for successful collaboration is most often of a non-financial
nature
ldquoThe ways that people work together shift over time which can affect your culture of
collaboration More important the introduction of collaboration technologies can also
change the culture of collaboration If handled properly the tools and the culture will
coevolverdquo
Dennis Kennedy
httpwwwclemensreijnennlpost20090220Application-Lifecycle-Management-and-
a-Collaborative-Cultureaspx
The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process
Life cycle model The life cycle model is a generic model It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development In the TMap life cycle model the test activities are divided across seven
phases Planning Control Setting up and maintaining infrastructure Preparation
Specification Execution and Completion Each phase is split up into a number of
activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan In the Control phase the activities in the test plan are executed monitored
and adjusted if necessary The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities The Preparation phase aims to have access to a test basis agreed with the
client of the test of adequate quality to design the test cases The tests are specified in
the Specification phase and executed in the Execution phase This provides insight into
the quality of the test object The test assignment is concluded in the Completion phase
This phase offers the opportunity to learn lessons from experiences gained in the project
Furthermore activities are executed to guarantee reuse of products
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsindexjsp
Business Driven Test Management
httpwwwclemensreijnennlpost20100628Business-Driven-Test-Management-with-
Microsoft-Test-Manageraspx
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results from design to deployment
Whether youre creating new solutions or enhancing existing applications Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologiesmdashincluding cloud and parallel computing
(from httpwwwmicrosoftcomvisualstudioen-usproducts2010-editionsultimate )
Test Lifecycle - Planning Phase
Planning phase of the Testing Lifecycle
The life cycle model is a generic model It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development In the TMap life
cycle model the test activities are divided across seven phases Planning Control
Setting up and maintaining infrastructure Preparation Specification Execution and
Completion Each phase is split up into a number of activities
In the Planning phase the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment This is laid down in
the test plan
This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality
More reading httpengtmapnetHomeTMapThe_4_essentialsindexjsp
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Work items
Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server In the central repository is the work items collected so everybody
knows each otherrsquos progress Every role every task has its own specific optimized tool
connected with this lsquosingle point of truthrsquo
The white circle just below the tools is the process template which supports drives a
predefined way of working
More reading httpmsdnmicrosoftcomen-uslibrarybb385832aspx
There are many different process templates which can be found in the MSDN landing
page and every organization has probably made some customizations In the box there
are the MSF Agile and the MSF CMMi
MSDN page httpmsdnmicrosoftcomen-usvstudioaa718795aspx
Read VSTS 2010 TMap Process Template 12 and 3
httpwwwclemensreijnennlpost20090703VSTS-2010-TMap-Process-Template-12-
and-3aspx
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 1 Create team projects
1 Open VS2010 and open the Team Explorer
2 check if the team explorer is connected to the Default Project Collection
3 add new team project
4 give it the name ltlt EmployeeApplication gtgt
5 select msf agile process template next next finish (an empty source tree will be
created)
6 after the team project is created add the sources to source control
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
7 open the application build and run it as a check
Do the same for the Tailspin application only after step 6 we have to set some things for
the database access Another dependency is the availability of MVC template in VS2010
see the USB
Every process template has artifacts artifacts which support the process the work of the
different roles and provide the team with metrics covering the quality of the process
See
Artifacts (Agile) httpmsdnmicrosoftcomen-uslibrarydd997580aspx
Artifacts (CMMI) httpmsdnmicrosoftcomen-uslibraryee332487aspx
Scrum 10 httpmsdnmicrosoftcomen-uslibraryff731587aspx
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
(from MSDN)
You can use work items to track and manage your work and information about your team
project A work item is a database record that Team Foundation uses to track the
assignment and progress of work You can use different types of work items to track
different types of work such as customer requirements product bugs and development
tasks
The work item type determines the workflow states and transitions The work item types
that are available in a project depend on the project template that the project uses Also
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment
Each work item represents an object that is stored in the Team Foundation database
Each work item is assigned a unique identifier a work item ID and type the work item
type Work item IDs are unique across all work item types in all team projects in a
project collection The work item type determines the work item fields that are available
for tracking information defaults defined for each field and rules and constraints
positioned on these fields and other objects that specify the work item workflow Every
change made to a work item field is stored in the work item log which maintains an
historical record of changes
You can create and modify work items by using Team Explorer Team System Web
Access Office Excel or Office Project When creating or modifying individual work items
you can work in the work item form by using Team Explorer or Team System Web
Access You can make bulk updates to many work items at a time by using Team System
Web Access Office Excel or Office Project
Work item types in the TMap for MSF Agile process template
User Story A team creates user stories to define the features functions and
requirements to be implemented
Task
Test Task
Test Basis Finding (see httptmapcodeplexcomwikipagetitle=Test basis finding)
Test Case
Bug You can track a code defect by creating a bug work item By creating a bug you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem
Issue You can define known or potential problems impediments or risks to your
project by creating issue work items
Shared Step Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager In shared steps you define a sequence of action and
validation steps to run as part of a test case
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
A user story communicates functionality that is of value to the end user of the product or
system
When you define a user story you must define the Title in the top section of the work
item form You can leave all other fields blank or accept their default values
Link types work items can be linked to each other each link has some specific
characteristics You can create work items for bugs tasks features requirements and
value propositions for your project and then create relationships among them that meet
your projects business goals
Work item types can have a specific flow in this example the flow of the user story WIT
Teams can change the flow specific their needs But be careful it must support the
system under development you can make it as crazy as you can image
The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor
More reading httpmsdnmicrosoftcomen-uslibrarydd380634aspx
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
An example the role of the tester and the usages of work items
During the planning phase of the project also called iteration 0 [first blue piece] user
stories are collected brainstormed defined hellip in VSTS this information is collected in
the work item type lsquouser storyrsquo
During the planning phase of the project also called iteration 0 [first blue piece] user stories
are collected brainstormed defined hellip in VS this information is collected in the new work item type lsquouser storyrsquo [image below]
During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab The new 2010 functionality of hierarchies between work items is used for this
Another task also executed during this phase is the creation of test cases Within the TMap
methodology this is described in the planning phase Where you create the test plan for that iteration based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area See the
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
initial work items for iteration 1 in the image below added during the unfolding of the TMap
process template
So not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration These test tasks are specific to testers like
lsquocreate test cases for area hellip based on the test technique lthellip decide based on risk
and customer contactgtrsquo
List of different test techniqueshellip
Data combination test (DCoT)
Data cycle test (DCyT)
Decision table test (DTT)
Elementary comparison test (ECT)
Error guessing (EG)
Error testing (ET)
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Process cycle test (PCT)
Real life test (RLT)
Semantic test (SEM)
Syntactic test (SYN)
Use case test (UCT)
So just like the implementation tasks also the test tasks are a link type have hierarchy with the user story [this is TMap process template specific]
We end up with a list of tasks for a user story implementation tasks and test taskshellip testers
and developers are going to execute these tasks together in parallel during the iteration we
have implemented sources for the user story and test cases which are ready for execution
This user story is finished when every implementation task is fulfilled all test cases are successful executed and hellip the tester hasnrsquot got any open tasks so all test cases are created
When giving the tester the team a place where they can record their testing tasks testing is
really going to be a first class citizen of the lifecycle beside this benefit the connection
between test activities risk user story and test cases gives a great opportunity for reports based on test effort risks and implementation later on more on thishellip
With this this lightweight addition to the user story work item we got a closer to several Agile Testing key characteristics for example
ndashgt just like the developers testers breakdown
there work discusing it with the customer
ndashgt and all tests are created
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks Within VSTS this is done in the
implementation tab of the user story Work item The new 2010 functionality of
hierarchies between work items is used for this
More reading httpwwwclemensreijnennlpost20090903Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-storiesaspx
A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project Tasks should represent a small
unit of work that can be accomplished within one to two days You can break larger tasks
down into smaller subtasks
You can create a task to track work to develop code design and run tests address bug
and perform regression testing In addition you can create tasks to support generic work
that must be performed
By tracking work hours for each task the team can gain insight into the progress that it
has made on the project
You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress identify problems in the flow of work and determine the team burn rate
More reading httpmsdnmicrosoftcomen-uslibrarydd380700aspx
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle Testers do not use their own separate technical
tools anymore but use integrated tools that are used by architects and developers
Effectively tearing down the wall between developers and testers
But good tools are not enough Also a clear separation of roles tasks and authorizations
are necessary Finally and most importantly a structured approach determines how
successful you are with your test strategy
For test tasks look at the activities in the different TMap phases
httpengtmapnetHomeTMapThe_4_essentialsStructured_Test_ProcessAcceptance
_and_system_testsPlanning_Phasejsp
More reading httpwwwclemensreijnennlpost20090421Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Editionaspx
A team uses test cases to define tests that will support testing of user stories You can
define manual test cases that specify a sequence of action and validation steps to run or
you can specify automated test cases that reference an automation file
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Areas and Iterations are typically artifacts used by development teams but now test is
connected with them we should have some knowledge about these topics
Areas are used to breakdown the application in functional pieces and iterations are used
to breakdown the effort of creating the application in time boxes
Often the testing effort is also broken-down in these parts
More reading
httpmsdnmicrosoftcomen-uslibrarydd997576aspx
httpmsdnmicrosoftcomen-uslibraryms181692aspx
Specific privileges are needed to create them
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 2 Create user stories and tasks
for the employee application
the user stories are used in the other labs
1 open you favorite TFS client (MTM VS2010 Web access )
2 add a user story and fill the necessary fields
3 add linked tasks
4 repeat several times
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Work Item Queries
With work items in place several queries and reports can provide valuable information
about the state of the project
Default there are already some powerful queryrsquos available to plan and find work
assigned to you
To access work item queries you must have the appropriate permissions
More reading httpmsdnmicrosoftcomen-uslibrarydd380734aspx
Show and run the queries for the previously executed exercise
Beside the default queries itrsquos also possible to create your own queries When a query is
saved in the lsquomy queriesrsquo folder they will show up in all other tools
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
More reading httpmsdnmicrosoftcomen-uslibraryms181352aspx
Work Item Query Language (Work Item Query Reference)
You can find bugs tasks other types of work items and links between work items by
using one of the WorkItemStoreQuery methods or a Query object These queries use
the work item query language (wiql) which resembles T-SQL
More reading
httpmsdnmicrosoftcomen-uslibrarybb130198(VS90)aspx
httpmsdnmicrosoftcomen-uslibraryms181352aspx
Beside the creation of flat lists queries itrsquos also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items Two different types of
queries can be created one based on link types and one as a tree view
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
you can perform the following tasks
Track and check features against value propositions
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time
Save changes to work items You can quickly save any changes that you make to
multiple work items within the list of results from the query
More reading httpmsdnmicrosoftcomen-uslibrarydd286632aspxFind
Type of query You can perform complex queries and view link associations between
work items with the addition of these two new types of queries
Work Items and Directed Links Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify You can
expand and collapse leaf nodes and within Team Explorer drag work items to change
link associations
Tree of Work Items Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links You can expand and collapse leaf nodes and
within Team Explorer drag work items to change link associations
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Reports Basic
Now we can collect the information with queries we need to visualize them Within TFS
this can be done with SQL Reports and Excel Reports First the default SQL Reports
Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions In the 2010 release wersquove enhanced every SQL Server
Reporting Services report Yoursquoll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box In 2010
wersquove also added new Excel Reports TFS feeds the reports via Work Item queries for fast
and flexible reporting Finally rich SharePoint dashboards enable quikc visualization of
project status In addition when used with SharePoint Server products you can publish
reports in a familiar and capable environment
Letrsquos look at the new Stories Overview report ltBuildgt
First you see the report contains a description describing what this report is showing
ltBuildgt
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Second you get a list of related reports that might help expanding on the data shown
here ltBuildgt
Finally you see the work progress on the user stories in context with data about test
results and bugs
Next as you scroll down you see additional information in the footer ltBuildgt
First on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report which has additional details including healthy and
unhealthy report examples ltBuildgt
Second you get the parameter values used to generate the report something that helps
when looking over printed copies in meetings for example ltBuildgt
And finally you get information on how up-to-date is this report Every report in the
2010 release is like this
As you can see the Quality dashboard has four main graphs ltBuildgt
First you can see if your test team is making progress on running test plans ltBuildgt
Second you can see how build are doing over time Whatrsquos the trend like Are you
having lots of success or failure ltBuildgt
Third whatrsquos your bug trend like Are you closing out bugs or are you stagnate Or is the
velocity of your bug filing far exceeding your teamrsquos ability to fix test and close out
bugs ltBuildgt
Finally are you seeing a bad trend related to bug reactivationsmdashbugs that were closed
reopened by test as not fixed All of this information is there for you in a quick heads up
dashboard format
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
More reading httpmsdnmicrosoftcomen-uslibrarydd380714aspx
While SQL Reports are hard to change are Excel reports much easier
More reading httpmsdnmicrosoftcomen-uslibrarydd997876aspx
For example a report based on the results of a flat list imported in
Excel
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports
Beside the default SQL reports there are also default Excel reports and workbooks
The default excel reports are
httpmsdnmicrosoftcomen-uslibrarydd997876aspx
You can use workbooks to build your product backlog and plan work by assigning the
user stories tasks bugs and issues to specific iterations also referred to as sprints By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v50 you can quickly create work items and modify the rank
priority state and assignments of multiple work items at the same time
httpmsdnmicrosoftcomen-uslibrarydd997891aspx
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Microsoft Test Manager
The not so standard user interface experience of MTM what to find where
httpmsdnmicrosoftcomen-uslibraryff458183aspx
(from Improving Developer-Tester Collaboration with
Microsoftreg Visual Studio reg 2010 httpchannel9msdncomshowsVS2010LaunchDoug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010 )
Microsoft Test Manager 2010 is for testers what Visual Studio is for developers That is to
say where Visual Studio is an IDE ndash an integrated development environment Test
Manager is an ITE ndash an integrated test environment This is the interface that a tester will
use to create test cases organize test plans track test results and file bugs when
defects are found Test Manager is integrated with Team Foundation Server and is
designed to improve the productivity of testers While I am not going to do a deep-dive
of all that Test Manager can do it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution and ensure that when a
tester files a bug it is actionable with very little work on the testerrsquos part
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Test Lifecycle ndash The Control
Phase I
Test Plans
Controlling the testing effort beside bug triage this also means test case management
what to run when what has been run etc
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end the test manager manages the test process optimally and reports on it
Within MTM test case can be organized within test plans suites and environmental
configurations How the test cases are organized depends on the type of organization
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
A test plan organization can be as simple or as complex as you want based on your
testing approach You might want to create a test plan for each sprint if you are using an
agile methodology Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories Or you might create a test plan for each
specific milestone if you are using another approach Another way is creating test plans
per test type
In Microsoft test manager a test plan holds all the information needed to maintain
organize and execute test cases It doesnrsquot hold information about the testing effort this
is the reason I talk about test execution plan
More reading httpmsdnmicrosoftcomen-uslibrarydd286583aspx
Without a test plan you canrsquot access test testing center in Microsoft Test Manager
After you created a test plan you can set its properties in
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Setting the state property to inactive will make the test plan disappear from the start
screen But it still can be accessed from another test plan in the lsquoorganizersquo section
Test plans should have a clean and descriptive title
More reading httpblogsmsdncombanuttharaarchive20100922guidance-for-
creating-test-plans-and-test-suitesaspx
httpwwwclemensreijnennlpost2009090902-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010aspx
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Test Configurations
You can assign a set of default configurations to your test plan that you want to cover for
quality purposes You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run
Some proposed configurations are the different environments people work with test
development etc
More reading httpblogsmsdncombvstsqualitytoolsarchive20091030test-
configurations-in-a-test-planaspx
Test Suites
You can create test suites in your test plan to group your test cases into suites based on
your needs There are three different suite types with different characteristics
1 Requirement bases suite You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement
Test cases added to this suite are automatically linked to the requirement and
test cases which are linked somewhere else (in visual studio another suite in
web acces) to the requirement are also visble in this suite
2 A folder like suite with no additional functionality other than hold the test suites
Folder Test suites can contain child suites (the other two not)
3 A query based suite which holds a collection of test cases with properties conform
the selection criteria
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
You canrsquot add test cases to this suite they are selected by the query
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 3 Create and configure test
plan
1 select add test plan in the black start screen and add a test plan
2 open the iteration test plan
3 open the test plan properties and edit the configurations
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3 add a new variable were we can but information about 64 bit and 32 bit machines
6 delete the current configurations and add or own client configuration
Windows 7 64 bit
Windows 7 32 bit
Windows 2008 R2 64 bit
7 set a default configuration for new test cases
Windows 7 64 bit
Windows 7 32 bit
8 save and close the test plan properties dialog
Add an additional end2end test plan
1 go to the lsquoorganizersquo section and add a new test plan
2 call it E2E Test plan and set several properties and configurations
Add Test suites to the iteration test plan
1 add requirement based test suite for the userstories
2 add query based test suite for regression candidateshellip prio 1
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3 add query based test suite for automation hellip automated status == planned or
automated
4 add a lsquonormalrsquo test suite for exploratory tests
TCMEXE
With the command line tool TCMexe you can get lists of test plans configurations and
more Also you can execute test cases with this command line tool
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
See httpmsdnmicrosoftcomen-uslibraryff942472aspx for listing test plans
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
With the collected ID you also can share links to test artifacts (like test plans) In the
lsquotest planrsquo start screen you can find on the top the lsquocopy URL for planrsquo button
Othere artifacts can be shared the same way
Test Lifecycle ndash Specification
Phase
After test planning and test control we get to the test specification phase in the testing
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
lifecycle
Aim
During the Specification phase the required tests and starting points are specified The
aim is to have as much as possible prepared in order to be able to run the test as
quickly as possible when the developers deliver the test object
Activities
Within the Specification phase the following activities are distinguished
Creating test specifications
Defining central starting point(s)
Specifying the test object intake
During the test case specification we create test cases with corresponding test steps
Shared steps and test data iterations are test specific items which need special attention
Test Cases
Test cases are work items with a specific tab where test steps can be defined These test
steps can only be edited from within MTM
You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional You can add test
cases to your test plan using Microsoft Test Manager
More information
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
httpmsdnmicrosoftcomen-uslibrarydd380712aspx
httpmsdnmicrosoftcomen-uslibrarydd286729aspx
Test case naming conventions discussion
What is the best way to name your test caseshellip
For quick editing of test cases (only the logical not the steps) you can use the team
foundation toolbar in excel to open a test query
A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step and both of them can contain additional information
(attachments)
See image three steps
1normal action step
2action step with validation
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3normal action step with attachment (validation action step can also contain an
attachment)
What kind of action steps do I need to use for my test case and how to write themhellip you
are probably not the one who executes the test case
more reading httpwwwclemensreijnennlpost20101029Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-useaspx
An easy way to share or re-use test steps is by copy paste the steps It works perfect to
and from excel
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Shared Steps
Your team can use shared steps to streamline definition and maintenance of manual test
cases Many tests require the same sequence of steps to be performed for multiple test
cases By creating shared steps you can define a sequence of steps once and insert it
into many test cases For example if each test case requires a tester to log on to the
application you can create a set of shared steps to perform these actions You can then
add the shared steps to each test case and run the steps using Test Runner
Candidates and reasons for shared steps
bull Reuse
ndash Many steps reused in many test cases Benefit time retyping boring
work
bull Expected changes
ndash for example environment variables change [URL] Benefit Time test
cases which need this step are updated automatically also action
recording
bull Pre- conditions
ndash bring the system under test to a certain state and execute the different
verifications Benefit beside time you are certain the application under
test is in the same state for every test case
bull Back to initial state
- after test execution bring the environment back to the initial state for
new test cases Benefit beside time you are certain the application under
test is always in a clean state for new test cases
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
There are somethings to think about and know before using shared steps
- Parameters defined in shared steps must also be defined in the test case for
every test data iteraiton
- Expected result in a shared step limits the re-use of the shared step Are you
testing the same thing over and over again
Test Data Iterations
If you want to run a manual test case multiple times with different data you do not have
to create multiple copies of the test case You can add parameters to the actions or
expected results for any test step in your test case Then you can add multiple sets of
values for the parameters that you want to use in your test Each set of values for the
parameters is run as an individual iteration of the test using Test Runner You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test
More reading httpmsdnmicrosoftcomen-uslibrarydd997703aspx
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
If you have configured test settings to create an action log for the test being run
selecting the Create action recording option will use the settings that you configured If
you did not configure the actions diagnostic data adapter in your test settings then the
default values are used to create your action recording
The same as with test steps test data can be copied and re-used in or from excel
Lab 4 Create test cases
For the employee application we are going to create test cases with test data iterations
and shared steps
Specify multiple test cases
1 open the iteration test plan
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
2 add a new test case to the requirements based test suite
3 give the test case a useful name
4 add the steps
1 open Employee application
2 fill lsquoClemensrsquo
3 fill lsquoReijnenrsquo
4 push lookup lsquowith expected resultsrsquo lsquo6913rsquo
5 close application
5 check if all the other fields are correct and save the test case
6 create some more test cases for the lsquoshow salaryrsquo functionality and employee not
found
Add test data iterations and parameters
1 open the test case which has been created during the first steps
2 change 6 fill lsquoClemensrsquo fill givename
7 fill lsquoReijnenrsquo fill surname
8 push lookup with expected results lsquo6913rsquo expected results phonenumber
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3 add the values
1 Clemens 6913 Reijnen
2 andre andre 1000
3 alf alf 2000
Create shared step and reuse it
0 not necessary but will make life easier create shortcut on the desktop for the
employee application executable (the executable can be found after compiling the
solution within VS2010 in the bin directory)
1 open the test case which has been created during the first steps
2 select the first step lsquoopen Employee applicationrsquo right click and select the context
menu item lsquocreate shared steprsquo
3 give it a name and save the shared step See that the step is now bold (optional
save and close the test case)
4 go to lsquoorganizersquo ndash lsquoshared steps managerrsquo select the shared step and click lsquocreate
action recordingrsquo
5 Test runner will open select lsquocreate action recordingrsquo and start the application
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
6 when finished save and close the Test Runner
7 repeat these steps for the lsquoclose applicationrsquo step
8 use the lsquoopenrsquo and lsquoclosersquo shared steps in the other test cases
Test Lifecycle ndash Execution
Phase
Aim
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
To obtain insight into the quality of the test object through the execution of the agreed
tests
Activities
Within the Execution phase the following activities are distinguished
Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results
Test Run
You can start as manual test run by selecting one or more test cases this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite and click run When the test run starts the test runner will guide the tester to the
test process
From msdn
Running manual tests using Test Runner can help you identify problems in your
application Problems that are typically identified when you run manual tests include
incorrect functionality or bugs poor performance and security and connectivity issues
The outcome resolutions and comments that you can enter during a manual test are
saved as test results
More reading httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Selecting the lsquorun with optionsrsquo menu will give the testers some more control over the
test settings and environments Also when the test case is an automated test case you
can use this menu item to run in manually
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
More info httpmsdnmicrosoftcomen-uslibrarydd286725aspx
Test Runner
The test runner has a minimalistic user interface but with a lot of options
- The return to testing center button opens microsoft test manager again but wonrsquot
stop the test This is great for small editrsquos
- In the drop down you can see the ammount of test cases and test data iteration in
this test run
- On the bottom you can show hide the recorded steps will executing the test case
- On the bottom in the recorded actions pane you can remove unnecessary
recordings for cleaning the action recording
- Steps with action recordings recognized by the orange bar can be re-played
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Besides using the test runner it is also possible to execute test runs with the TCMexe
command line tool Making it easy to automated test execution during builds
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 5 Execute the created test
cases
For the employee application we are executing the create test cases
1 go to lsquorsquoTestrdquo ndash lsquorun testsrsquo and select either a test suite or a single test case and click
lsquorunrsquo
2 test runner will open check the lsquocreate action recordingrsquo and start the test Follow the
steps and mark the steps passed (if succesful)
Optional run them from TCMexe
1 open Visual Studio command prompt type lsquotcmexe run helprsquo and follow the
guidance
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Test Case Execution Results
The most important thing after execution of test cases is reporting about the results
what is the quality of my system
In Team Foundation Server there are many ways to get the needed information All the
data is stored in a data warehouse available for all the different client tools
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Test run results in MTM
In Microsoft test manager the reporting about the test runs is minimal you can see the
result of the test runs if they failed or passed
After you run tests you can review your test results to see which tests passed and which
tests failed For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results assign a reason for a failed test and assign a resolution
You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports
As you analyze the test results for each test you can select a failure type and resolution
for each failed test or add a comment about your analysis to this point To help with
your analysis you can open the test result for each test to see all previous runs of that
test
If you determine that there is an issue you can create a bug and associate the data from
the test result for this test with the bug If you determine that the failed test is caused by
a known issue you can link that test to an existing bug When this bug is fixed the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed
If you want to add data to the test run you can add attachments or comments to the
test run itself
More readings httpmsdnmicrosoftcomen-uslibrarydd553274aspx
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite A test point is the pairing of a test case with a test
configuration in a specific test suite If you have multiple configurations that you want to
use when you run your test you have a test point for each pairing of the test with a
configuration
More readings httpmsdnmicrosoftcomen-uslibraryee843861aspx
Also in the test plan properties window you can view the results But for most
organizations all this test result information isnrsquot enoughhellip to say something about the
quality of the system
Share the test run with peers by using a URL and the runID
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Because Test cases are Work item we can create queries to analyze the work The
defaults arehellip
Also default reports and excel reports are available
(show)
Browse through the created reports and discus
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
There are also pivot capabilities for the test artifacts Selecting the test fields show them
One important thing to notice is are the fields which donrsquot belong to a work item these
are MTM specific artifacts save to the datawarehouse
The organization looks like this Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server Not accessible together only in de olap
cubes
Some specific useful fields in the warehouse for reportings
(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server you can view just the measures dimensions and attributes
that pertain to reporting on tests results and test runs For example you can use these
measures to determine the overall quality of each build the tests that a particular build
affected and the number of test cases that were run You can also answer questions
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
about changes to the result outcomes
The Test measure group is based on the Test Results relational table which enables
reporting on test results as either a property of the tests or an independent outcome For
more information see Test Result Tables
More reading httpmsdnmicrosoftcomen-uslibraryms244708aspx
It is still a bit complicated or better a bit confusing how the different parts within
Microsoft Test Manager are organized This image is a try to clear things a bit
More reading httpwwwclemensreijnennlpost20100719Microsoft-Test-Manager-
Test-Artifact-organizatione280a6aspx
Test Lifecycle ndash Control Phase
II
Actions a test organization does during the control is not only the management of the
testing effort but also the management of bugs
Aim
Providing the client with sufficient insight into and the opportunity to influence the
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
following
The progress of the test process
The quality and risks of the test object
The quality of the test process
File a Bug
Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed This improves the quality of the applications that you are
testing The more information that you can provide with a bug the more quickly it can be
recreated and fixed Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create when you run your tests You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
When you run a test using the Test Runner
From Microsoft Test Manager when you view a test result
From Microsoft Test Manager when you view your bugs
You can also submit bugs from Microsoft Visual Studio 2010
All these bugs can be viewed updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010
Important topics for understanding and fully use the capabilities of MTM with bug filling
are
Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
File a Bug
File an Exploratory Bug
Verify a Bug
Collect Data on a Local Machine by Using Default Test Settings
If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application you can use the default test settings
More reading
httpmsdnmicrosoftcomen-uslibraryee231892aspx
httpblogsmsdncombamit_chatterjeearchive20090411diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collectoraspx
File a bug from Test Runner
When you run a manual test and discover a defect in the application under test you can
submit a bug directly from Test Runner without having to switch to Visual Studio
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug You can also update an existing bug if you discover
additional defects while you are running a test
More reading
httpmsdnmicrosoftcomen-uslibrarydd286746aspx
httpmsdnmicrosoftcomen-uslibrarydd293543aspx
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Submitting Bugs When You Run Your Manual Tests You can submit a bug immediately
when you run a test from Test Runner Your test steps and any information that you
have collected are automatically added to the new bug that you create
Submitting Bugs from Your Test Result You can submit a bug from Microsoft Test
Manager when you view a test result For example when you analyze a test result and
you determine that the test failed because of a product issue You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs
Submitting Bugs from Microsoft Visual Studio You can submit bugs using Microsoft
Visual Studio 2010 at any time
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Verify a bug
Using Microsoft Test Manager you can track the status of both bugs that are assigned to
you and bugs that you created You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not You can assign the bug to another user to provide more information decide to fix
the bug or determine whether it is actually a bug
When a bug is fixed it can be assigned to the tester to verify that the bug has been
fixed In the Verify Bugs view you can view bugs that are assigned to you and have to
be retested as shown in the following illustration Or you can create a custom query to
track all bugs that have to be retested
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 6 file a bug
Create a build
We need a build for later use
1 open Visual Studio
2 open lsquoTeam Explorerrsquo and select lsquobuilsrsquo right mouse click and select the context menu
lsquonew build definitionrsquo
rsquo
3 trigger lsquomanualrsquo
4 build defaults set the drop folder
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
5 process lsquoitems to buildrsquo the employee app solution
6 try the build by selecting lsquoqueue new buildrsquo
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
7 check if successful (Green bar)
When the build doesnrsquot run check the build service in the team foundation console
File a bug
1 set in the test settings local test run the wanted data collectors on
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
2 add to the first create test case (in lab 4) the test data iteration values 111
3 run the test case (it will crash at 111 -)
4 file a bug by selecting lsquocreate bugrsquo
5 give the bug work item a useful name ndash assign it to yourself save and close it
Resolve a bug
1 open Visual Studio and the employee application
2 open the Team Explorer and open the team query lsquomy Bugsrsquo look for the previously
filed bug
3 go to the solution explorer open the lsquoemployeescsrsquo file and look for the sentence with
lsquoelse if (firstnameEquals(1) ampamp lastnameEquals(1))rsquo
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Remove the lines
Exception ex = new Exception(Went something wrong)
throw ex
and the before the two below
Should look like this afterwards
else if (firstnameEquals(1) ampamp lastnameEquals(1))
Employee emp = new Employee(1 1 1sogetinl 1 1 1 1)
return emp
4 select the lsquoemployeescsrsquo file right click and select lsquocheck inrsquo
5 in the check in dialog select lsquowork itemsrsquo and check the bug select for action
lsquoresolversquo
6 queue a new build and wait till it finishes successful
Verify a bug
1 open Microsoft test manager
2 go to lsquotestrsquo ndash lsquoverify bugsrsquo select the resolved bug and click verify
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3 test runner will open Run the test casehellip and check if the bug is solved
4 when all steps are finished and marked passed (where necessary) close the test
runner MTM will ask if you want to close the bug
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Bug Reports
Monitor bug activity reactivations and trends You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them
Test Impact
By using Test Impact Analysis during code development you can identify the methods in
a test project that have been affected by code changes in your managed code solution
At each build of the solution on the local computer Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods You can then run the tests from the Test Impact View
window You can also use the Test Impact View window to identify and run any test
method that affects a particular code method
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
demo
By using test impact analysis you can identify the tests that you should run based on
coding changes that were made to the application that you are testing between builds
When you use test impact analysis with Microsoft Test Manager you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested Test impact information is gathered
only on tests with a status of passed It is not collected when you file a bug or when a
test that is marked as failed is complete
The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly To use
the Test Impact View in Microsoft Visual Studio 2010 you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 7 test impact
Attach build to test plan
1 open MTM
2 go to lsquoplanrsquo ndash lsquopropertiesrsquo and select the build filter created in the previous lab
(no need to set the quality)
3 select lsquobuild in usersquo select the latest build and assign it to the plan
Execute tests
1 execute several test cases
Make code change and kickoff a build
1 as in lab 7 lsquoResolve a bugrsquo hellip open the lsquoemployeescsrsquo file
2 remove the line
Employee emp = new Employee(Clemens Reijnen ClemensReijnenSogetinl 69
13 12000 Amstelveen Noord Holland)
And remove the from this line
Employee emp = new Employee(1234 4567 12344567Sogetinl 6913 1200
0 Amstelveen Noord Holland)
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
3 as in lab 7 lsquoResolve a bugrsquo hellip check in the changes and kickoff a build
Attach new build to test plan and evaluate impacted tests
1 go to lsquoplanrsquo ndash lsquopropertiesrsquo and assign the new build number to the plan
2 review the lsquoRecommended Testsrsquo and set them back to lsquoactiversquo
Create test from bug
An often overlooked feature in Microsoft Test Manager is the lsquocreate test from bugrsquo option
in the lsquoverify bugsrsquo section It sounds indeed a bit strange probably there is already a
test case in place which found this bug But this create test case feature has some
interesing scenariorsquos
When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps you can very easily create a well written tests case from
such a action recording
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
When you run an exploratory test and find a bug in the application you can submit an
exploratory bug directly from Test Runner Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug The actions are
recorded in an action log which can be used to create a manual test case
More reading
httpmsdnmicrosoftcomen-uslibraryee803840aspx
Other scenariorsquos for this feature is for example an application with no tests defined in
MTM you can let the business guy run his most important process while the test runner
is recording You can use this recoriding to create well written test cases Another
interesting scenario are migration projects when you want to migrate an application
from VB6 to NET for example record the most important test cases and create the test
cases
Lab 8 Create an exploratory bug and create a test case from it 1 open MTM
2 create a test case with the name expolritory test
3 add the step lsquostart exploratory testing for 5 minutesrsquo save and close it
4 run the test and select after some clicking around lsquocreate exploratory bugrsquo
5 go to lsquoverify bugrsquo select the bug right click and select lsquocreate test case from bugrsquo in
the context menu
6 look at the test steps
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Test Lifecycle - Infrastructure
Phase
Before moving on to the completion fase we need to setup or infrastrcuture first
Aim
To provide the required test infrastructure which is used in the various TMap phases and
activities
Activities
The Setting up and maintaining infrastructure phase consists of the following activities
Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
The lsquogreenrsquo Lab Center is the place to manage your test infrastructure you can manage
physical and vertualized test environments from this center
Where can I run the tests
There are different ways to execute tests the main difference is between the
environment types and technology used for it within Visual Studio 2010
One major difference is the provisioning of clean environments
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
On Physical Environments with Microsoft Test Manager
There are multiple ways to execute your test automation effort you can run them from and
on Microsoft Test Manager from and on Visual Studio during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or hellip many flavors
so where should you execute your automated tests and how should you configure your test infrastructure
Where to run your automated tests and how to configure the test infrastructure
The different flavors
Flavor A Execution from
VS2010hellip
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Purpose Real developer test test automation dry
run
A valid scenario is when Developers create a
codedUI test from a test case which found a bug
To reproduce the bug developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug
Triggered by right mouse click ndash run tests in test project or Test List Editor run checked tests
Information httpmsdnmicrosoftcomen-uslibrarydd286580aspx
Pro
Easy setup
Debug-able test automation No additional configuration needed
Con
No collection of test results in TFS Test Manager Run on developer environment
Data Diagnostic Settings in the LocalTestsettings file are configurable
Flavor B Execution during
build with Build controller
Purpose Part of Build Verification Tests (its
strange to run UI tests on a build and have a build service running in interactive mode)
Triggered by build
Information
Set the Build Service to run in interactive mode
httpblogsmsdncombmathew_aniyanarchive
20090526coded-ui-test-in-a-team-buildaspx
Configure the build to run the tests
httpmsdnmicrosoftcomen-
uslibraryms182465aspxCreateBuildType
Pro
Easy setup Test results in build result
Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart) Tests are executed on build environment
Flavor C Execution during build with Test controller
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Purpose Part of Build Verification Tests distribute
tests over multiple test agents (Preferred configuration above flavor B)
Triggered by build
Information
Configure Test Controller (donrsquot register it with a
project collection )
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestControllers
Configure
Test Agents on clients (interactive mode)
httpmsdnmicrosoftcomen-
uslibrarydd648127aspxTestAgents
Configure Testsettings file in solution to use Test
Controller and Test Agents Configure the build to run the tests (see B)
Pro
Tests run distributed over multiple environments Test Results in Build result
Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments Test Settings from VS
Flavor D Execution from
Microsoft Test Manager
Purpose Part of Regression Tests (other type of test than BVT)
Triggered by MTM user right mouse click on test case run
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo physical environment
httpmsdnmicrosoftcomen-
uslibraryee390842aspx
httpmsdnmicrosoftcomen-
uslibrarydd293551aspx
Associate CodedUI test with WI Test Case from VS
httpwwwrichard-banksorg201011how-to-use-codedui-tests-watin-and-mtmhtml
Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM Full control by the tester
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure Hard to see which test case is automated
Flavor E Execution from MTM during Buildhellip
Purpose Part of BVT Preferred configuration
above flavor C Flavor D and E can be configured together
Triggered by Build
Information
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test lsquoagentrsquo environment
Associate CodedUI test with WI Test Case from VS
Create Build task to run TCM or MSTEST task for
Test Plan
httpblogsmicrosoftcoilblogsshairarchive201
01030how-to-run-coded-ui-tests-from-
command-lineaspx
How to Run Test Cases with Automation from the
Command Line Using Tcm
httpmsdnmicrosoftcomen-uslibrarydd465192aspx
Pro
Test run distributed over test environments
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build Test Settings from MTM
Con
Hard to configure maintenance of TCM commands in build
Enough flavors to choose fromhellip the pro and cons I wrote down can differ per situation My
preferred flavors are E and D configured together The reason test settings and test results
are where the belong in the testers hands And the TCM command line gives much more flexibility what to run where during the build
Lab Management for virtual environments has different configurations pros and conshellip
will make a post for this laterhellip
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky itrsquos a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments one drawback from flavor E will be
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
minimized when using this template the maintenance of the TCM commands
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 9 Install and configure the test
controller and test agent
httpmsdnmicrosoftcomen-uslibrarydd648127aspx
httpmsdnmicrosoftcomen-uslibraryff469838aspx
Lab 10 Setup Flavor D Automated Test Execution from Microsoft Test Manager
Configure the test controller that it is associated
with the team project collection
Create a physical environment Verify in the
controllers menu that there is an environment
connected with a test agent
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
In the test plan properties configure the
automated test settings
To add an automation script to a test case we
must open visual studio and add a test script to
the test case (see CodedUI test in this
document for the details)
Now your test infrastructure is ready to execute
automated tests from MTM Select run on a test
case with an automation script
Test Lifecycle - Completion
Phase
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test
Activities
The Completion phase consists of the following activities
Evaluating the test process
Preserving the testware
Test automation with Visual Studio
2010
Different test levels in which phase should we use test automation
What is what level and what do you do at that level
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
VS2010 has many technologies and capabilities for test automation
What is what
hellip
hellip
hellip
hellip
Within the different levels different technologies are interesting
Why hellip
In what phase of the project and by whom is the technology used
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Who en whenhellip
hellip
hellip
Why do you want to automate it with that technology at that phase what do you want to
accomplish
In this guid we focus on the UI automation capabilities of VS2010 How to use it why by
whom and how does the infrastructure looks like
When is test automation the
investment worth
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case But this isnrsquot without investment There are still some additional things to do which take time and money before we get the benefit of automated execution of a test case
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
These are some very basic test automation investment levels Just to keep in mind when working with Microsoft Test Manager and CodedUI
VS2010 ALM has several automation capabilities The most noticeable is CodedUI C code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner Fast Forward till a
validation step for manual validation This Fast Forward functionality is also valuable in the
shared steps and for test data iterations
The investment levels explained
Zero time investment completely no investment in any kind of automation
no Fast Forward no CodedUI Although the FF is recorded for every test it isnrsquot used MTM is
used in this situation not for its automation capabilities but more for its test case management bug filling and reporting capabilities
Which test cases donrsquot need any kind of automation depends The most logic reason would be
that the test case only is going to be executed once the other reason can be that automating the test case would be a to big investment and to complex and there are more reasons
This post is a useful source ldquoeffort-estimation-for-test-automationrdquo to make your decision to
automate or not other good reading is this pdf ldquoTEST AUTOMATION EFFORT ESTIMATION - Best practicesrdquo
Little time investment executing the same test case on multiple
environments or test cases executed several times during the sprint are candidates for
automation but sometimes the investment of making it CodedUI is too big Making use of the
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Microsoft Test Runnerrsquos Fast Forward capabilities will speed up test execution but getting
that additional benefit of it you have to tune the action recording to make the Fast Forward functionality in the Test Runner more smooth and less error prone
When executing a manual test case you never do it correct the first time You have to search
a bit maybe make a wrong step go back in the application etchellip execute a test script
complete correct with the optimal amount of clicks the first time is hard And when you do
click everything correct the first time the action recording will collect a bit too much
information in some situation (see image opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test) Cleaning the action
recording will make the FF more efficient and less error prone this cleaning of the action recording needs to be done during the execution of the test case in Microsoft Test Runner
Note The little investment of cleaning the action recording also is for the shared step and
test cases with test data iteration Test case which use test data iterations are candidate for a
bit more investment for a clean action recording shared steps are for sure candidates
Some more investment with Basic CodedUI functionality Although it is
very easy to create CodedUI C files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step) It is still a bigger
in time investment to create them Beside the time it takes to create the CodedUI C it also
asks an investment in test infrastructure to execute them see this post ldquoRunning Automated Tests on Physical Environments the different flavorshelliprdquo
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
See image you also have to create the assertion for the automated manual test case 1) the
manual validation in the Test Case 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion is added to the CodedUI test
The main idea for this investment level is that only the basic generated code is used (not customized) and an assertion is manual added to it
Serious investment in time with advanced CodedUI customization of
the CodedUI and UIMap Customizing the generated CodedUI is a real trap You can make
the most sophisticated UI tests in the world Resulting in C code which is even more
complicated as the functionality it tests with probably even as many or more bugs who tests
the tests trap So this investment should be done careful but sometimes itrsquos a worthy
investment For example for test cases which are going to be run the whole life of the system or maybe are boring and have many steps (see the links in the zero investment level)
Customizations of this level can be started from the lsquoUIMap builderrsquo (see image) change the
search properties move steps out of the code generation and customize them add fancy test data iteration to the steps etc etchellip
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 13 Create CodedUI
Create CodedUI from Action Recording
Add a CodedUI test to the test project in the
solution Select lsquouse excisting action recordingrsquo
In the work item picker dialog select a test case
with a valid action recoding Generate the C
Look at the generated methods and compare
them with the test steps in the test case Run
the CodedUI test
Add Assertion
All steps are generated except the assertion we
need to create generate this to make the
CodedUI test complete
Right click in the CodedUI code where you want
to add the assertion and select lsquoUse CodedUI test
Builderrsquo In the lower right corner of your screen
a little dialog appears you can use the crosshair
to select the control you want to validate
Generate the assertion and run the CodedUI
again
Create a CodedUI from scratch by using the recording option
Record action for CodedUI
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose
Methods can be moved controls renamed and its
search criteria tuned
The search configuration can be edited from the
properties menu
httpmsdnmicrosoftcomen-
uslibraryff398062aspx
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven run them more as once with different
datasets These datasets can come from a test
case or other data source like excel and
databases
httpmsdnmicrosoftcomen-
uslibraryee624082aspx
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 14 Execute the CodedUI during the build from Visual Studio
Execute codedUI tests from Visual Studio within a build
Setting up Flavor C
For this lab the test controller isnrsquot connected to
a team project collection we are going to
execute the test case from Visual Studio
In the test settings file we must configure the
roles where the test will be execute
httpmsdnmicrosoftcomen-
uslibraryee256991aspx
there has to be a test controller agent
infrastructure in pace
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Define a build and set the test properties to use
the test settings file
Queue the build and watch the test agent
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 16 Execute the codedUI from
MTM
Setting up Flavor D
The test controller must be associated with the
team project collection
Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Lab 17 Execute the codedUI during
the build from MTM
Setting up Flavor E
You need two environments with test agents for this lab I use a vpc and its host for it
With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed In this lab we will configure the build
so test cases are executed on these specific
environments
Default the configuration of a test controller is
for load run automated test cases on all agents
so we execute test cases as fast as possible
When you want to run a test case on a specific
environment we need to do some additional
things
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
We will execute the test cases from the
command line tool tcmexe
Get the necessary information And compose the
command lines with parameters Look at the
query in step 2 where we select the test cases
Open the build defeinition workflow xaml and
add two lsquoinvokeprocessrsquo actions where we call
the tcmexe tool with its parameters (note the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production this is only for demo purpose)
Queue the build and watch the test agents
On Virtualized Environments with
Lab Management and SCVMM
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
For a complete guid about Virtulized Test environments see the Rangers Lab Guide
httpralabmancodeplexcom
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
From httpralabmancodeplexcom
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Extensibility
Tools
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Custome Data Adapter
See lab guid rangers
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Azure and test automation httpwwwclemensreijnennlpost20110329VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-developmentaspx
Test Manager and an agile
process httpwwwclemensreijnennlpost20100921Agile-Test-practices-with-Microsoft-
Visual-Studio-2010aspx
Intelitrace
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Appendix
Proposed training agenda 2 day
training
Student and trainer Environment
Student and trainer Environment