Software Testing

Anuradha Bhatia 1

TESTING TOOLS AND MEASUREMENT CONTENTS

I. Limitation of Manual Testing and Need for Automated Testing Tool.

II. Features of Test Tool: Guidelines for Static and Dynamic Testing Tools.

III. Advantage and Disadvantage of Using Tools.

IV. Selecting a Testing Tool.

V. When to Use Automated Test Tools, Testing Using Automated Tool.

VI. What are Metrics and Measurements: Types of Metrics, Project Metrics, Progress and Productivity Metrics?

Software Testing

Anuradha Bhatia 2

I. Limitation of Manual Testing and Need for Automated Testing Tool.

(Question: Explain the need of Automation testing over manual testing. – 8 Marks)

i. Manual testing is slow and costly. ii. It is very labor intensive, it takes a long time to complete tests.

iii. Manual tests don’t scale well. As the complexity of the software increases the complexity of the testing problem grows exponentially. This leads to an increase in total time devoted to testing as well as total cost of testing.

iv. Manual testing is not consistent or repeatable. Variations in how the tests are performed as inevitable, for various reasons. One tester may approach and perform a certain test differently from another, resulting in different results on the same test, because the tests are not being performed identically.

v. Lack of training is the common problem, although not unique to manual software testing.

vi. GUI objects size difference and color combinations are not easy to find in manual testing.

vii. Not suitable for large scale projects and time bound projects. viii. Batch testing is not possible, for each and every test execution Human user

interaction is mandatory. ix. Comparing large amount of data is impractical. x. Processing change requests during software maintenance takes more time.

NEED of automated testing

i. An automated testing tool is able to playback pre-recorded and predefined actions, compare the results to the expected behavior and report the success or failure of these manual tests to a test engineer.

ii. Once automated tests are created they can easily be repeated and they can be extended to perform tasks impossible with manual testing.

iii. Because of this, savvy managers have found that automated software testing is an essential component of successful development projects.

1. Automated Software Testing Saves Time and Money

i. Software tests have to be repeated often during development cycles to ensure quality. Every time source code is modified software tests should be repeated.

ii. For each release of the software it may be tested on all supported operating systems and hardware configurations.

iii. Manually repeating these tests is costly and time consuming. Once created, automated tests can be run over and over again at no additional cost and they are much faster than manual tests.

iv. Automated software testing can reduce the time to run repetitive tests from days to hours.

v. A time savings that translates directly into cost savings.

Software Testing

Anuradha Bhatia 3

2. Testing Improves Accuracy

i. Even the most conscientious tester will make mistakes during monotonous manual testing.

ii. Automated tests perform the same steps precisely every time they are executed and never forget to record detailed results.

3. Increase Test Coverage

i. Automated software testing can increase the depth and scope of tests to help improve software quality.

ii. Lengthy tests that are often avoided during manual testing can be run unattended.

iii. They can even be run on multiple computers with different configurations. iv. Automated software testing can look inside an application and see memory

contents, data tables, file contents, and internal program states to determine if the product is behaving as expected.

v. Automated software tests can easily execute thousands of different complex test cases during every test run providing coverage that is impossible with manual tests.

vi. Testers freed from repetitive manual tests have more time to create new automated software tests and deal with complex features.

4. Automation Does What Manual Testing Cannot

i. Even the largest software departments cannot perform a controlled web application test with thousands of users.

ii. Automated testing can simulate tens, hundreds or thousands of virtual users interacting with network or web software and applications.

5. Automated QA Testing Helps Developers and Testers

i. Shared automated tests can be used by developers to catch problems quickly before sending to QA.

ii. Tests can run automatically whenever source code changes are checked in and notify the team or the developer if they fail.

iii. Features like these save developers time and increase their confidence.

6. Team Morale Improves

i. This is hard to measure but we’ve experienced it first hand, automated software testing can improve team morale.

ii. Automating repetitive tasks with automated software testing gives your team time to spend on more challenging and rewarding projects.

iii. Team members improve their skill sets and confidence and, in turn, pass those gains on to their organization.

Software Testing

Anuradha Bhatia 4

7. Test Complete is a Powerful and Affordable Automated

Software Testing Tool

i. Test Complete addresses a full range of software testing challenges facing corporate IT departments, product developers, QA engineers, and consultants.

ii. Test Complete enhances the software testing process by increasing efficiency, removing complexity and lowering costs

ADVANTAGES

Automated Testing Manual Testing

• If you have to run a set of tests repeatedly automation is a huge gain

• If Test Cases have to be run a small number of times it's more likely to perform manual testing

• Helps performing "compatibility testing" - testing the software on different configurations

• It allows the tester to perform more ad-hoc (random testing)

• It gives you the ability to run automation scenariosto perform regressions in a shorter time

• Short term costs are reduced

• It gives you the ability to run regressions on a code that is continously changing

• The more time tester spends testing a module the grater the odds to find real user bugs

• Can be run simultaneously on different machines thus decreasing testing time

• Long term costs are reduced

DISADVANTAGES

Automated Testing Manual Testing

• It's more expensive to automate. Initial investments are bigger than manual testing

• Manual tests can be very time consuming

• You cannot automate everything, some tests still have to be done manually

• For every release you must rerun the same set of tests which can be tiresome

OTHER FACTORS

• The performance of your test tools

• The knowledge level of your testing team

• The continuous growth of software to be tested

• Number of necessary regressions

II. Features of Test Tool: Guidelines for Static and Dynamic Testing Tools.

1. Static Testing Tool

(Question: Explain the static testing tool as a test tool feature. – 4 Marks)

Software Testing

Anuradha Bhatia 5

i. Static analysis tools are generally used by developers as part of the development

and component testing process. The key aspect is that the code (or other artefact)

is not executed or run but the tool itself is executed, and the source code we are

interested in is the input data to the tool.

ii. These tools are mostly used by developers.

iii. Static analysis tools are an extension of compiler technology – in fact some

compilers do offer static analysis features. It is worth checking what is available

from existing compilers or development environments before looking at

purchasing a more sophisticated static analysis tool.

iv. Other than software code, static analysis can also be carried out on things like,

static analysis of requirements or static analysis of websites (for example, to

assess for proper use of accessibility tags or the following of HTML standards).

v. Static analysis tools for code can help the developers to understand the structure

of the code, and can also be used to enforce coding standards.

2. Dynamic Testing Tool

(Question: Explain the dynamic testing tool as a test tool feature. – 4

Marks)

i. Dynamic analysis tools are ‘dynamic’ because they require the code to be in

a running state. They are ‘analysis’ rather than ‘testing’ tools because they

analyze what is happening ‘behind the scenes’ that is in the code while the

software is running (whether being executed with test cases or being used in

operation).

ii. Let us take an example of a car to understand it in a better way. If you go to a

showroom of a car to buy it, you might sit in the car to see if is comfortable

and see what sound the doors make – this would be static analysis because

the car is not being driven. If you take a test drive, then you would check that

how the car performs when it is in the running mode e.g. the car turns right

when you turn the steering wheel clockwise or when you press the break then

how the car will respond and can also check the oil pressure or the brake fluid,

this would be dynamic analysis, it can only be done while the engine is running.

Features or characteristics of dynamic analysis tools are as follows:

i. To detect memory leaks;

Software Testing

Anuradha Bhatia 6

ii. To identify pointer arithmetic errors such as null pointers;

iii. To identify time dependencies.

Eventually when your computer’s response time gets slower and

slower, but it get improved after re-booting, this may be because of

the ‘memory leak’, where the programs do not correctly release blocks

of memory back to the operating system. Sooner or later the system

will run out of memory completely and stop. Hence, rebooting restores

all of the memory that was lost, so the performance of the system is

now restored to its normal state.

These tools would typically be used by developers in component

testing and component integration testing, e.g. when testing

middleware, when testing security or when looking for robustness

defects.

Another form of dynamic analysis for websites is to check whether

each link does actually link to something else (this type of tool may be

called a ‘web spider’). The tool does not know if you have linked to the

correct page, but at least it can find dead links, which may be helpful.

III. Advantage and Disadvantage of Using Tools. (Question: Give advantages and disadvantages of using automated tools. – 4

Marks)

There are many benefits that can be gained by using tools to support testing.

i. Reduction of repetitive work: Repetitive work is very boring if it is done

manually. People tend to make mistakes when doing the same task over

and over. Examples of this type of repetitive work include running

regression tests, entering the same test data again and again (can be done

by a test execution tool), checking against coding standards (which can be

done by a static analysis tool) or creating a specific test database (which can

be done by a test data preparation tool).

ii. Greater consistency and repeatability: People have tendency to do the

same task in a slightly different way even when they think they are

repeating something exactly. A tool will exactly reproduce what it did

before, so each time it is run the result is consistent.

Software Testing

Anuradha Bhatia 7

iii. Objective assessment: If a person calculates a value from the software or

incident reports, by mistake they may omit something, or their own one-

sided preconceived judgments or convictions may lead them to interpret

that data incorrectly. Using a tool means that subjective preconceived

notion is removed and the assessment is more repeatable and consistently

calculated. Examples include assessing the cyclomatic complexity or nesting

levels of a component (which can be done by a static analysis tool),

coverage (coverage measurement tool), system behavior (monitoring tools)

and incident statistics (test management tool).

iv. Ease of access to information about tests or testing: Information

presented visually is much easier for the human mind to understand and

interpret. For example, a chart or graph is a better way to show

information than a long list of numbers – this is why charts and graphs in

spreadsheets are so useful. Special purpose tools give these features

directly for the information they process. Examples include statistics and

graphs about test progress (test execution or test management tool),

incident rates (incident management or test management tool) and

performance (performance testing tool).

Disadvantage

i. Unrealistic expectations from the tool: Unrealistic expectations may be

one of the greatest risks to success with tools. The tools are just software

and we all know that there are many problems associated with any kind of

software. It is very important to have clear and realistic objectives for what

the tool can do.

ii. People often make mistakes by underestimating the time, cost and effort

for the initial introduction of a tool: Introducing something new into an

organization is hardly straightforward. Once you purchase a tool, you want

to have a number of people being able to use the tool in a way that will be

beneficial. There will be some technical issues to overcome, but there will

also be resistance from other people – both need to be handled in such a

way that the tool will be of great success.

iii. People frequently miscalculate the time and effort needed to achieve

significant and continuing benefits from the tool: Mostly in the initial

phase when the tool is new to the people, they miscalculate the time and

Software Testing

Anuradha Bhatia 8

effort needed to achieve significant and continuing benefits from the tool.

Just think back to the last time you tried something new for the very first

time (learning to drive, riding a bike, skiing). Your first attempts were

unlikely to be very good but with more experience and practice you

became much better. Using a testing tool for the first time will not be your

best use of the tool either. It takes time to develop ways of using the tool

in order to achieve what is expected.

iv. Mostly people underestimate the effort required to maintain the test

assets generated by the tool: Generally people underestimate the effort

required to maintain the test assets generated by the tool. Because of the

insufficient planning for maintenance of the assets that the tool produces

there are chances that the tool might end up as ‘shelf-ware’, along with

the previously listed risks.

v. People depend on the tool a lot (over-reliance on the tool): Since there

are many benefits that can be gained by using tools to support testing like

reduction of repetitive work, greater consistency and repeatability, etc.

people started to depend on the tool a lot. But the tools are just a software

they can do only what they have been designed to do (at least a good

quality tool can), but they cannot do everything. A tool can definitely help,

but it cannot replace the intelligence needed to know how best to use it,

and how to evaluate current and future uses of the tool. For example, a

test execution tool does not replace the need for good test design and

should not be used for every test – some tests are still better executed

manually. A test that takes a very long time to automate and will not be

run very often is better done manually.

IV. Selecting a Testing Tool. (Question: Explain the factors considered for selecting the testing tool. – 4 Marks)

i. While introducing the tool in the organization it must match a need within

the organization, and solve that need in a way that is both effective and

efficient.

ii. The tool should help in building the strengths of the organization and should

also address its weaknesses.

Software Testing

Anuradha Bhatia 9

iii. The organization needs to be ready for the changes that will come along

with the new tool.

i. If the current testing practices are not good enough and the organization is

not mature, then it is always recommended to improve testing practices first

rather than to try to find tools to support poor practices. Certainly, we can

sometimes improve our own processes in parallel with introducing a tool to

support those practices and we can always pick up some good ideas for

improvement from the ways that the tools work.

ii. Do not depend on the tool for everything, but it should provide support to

your organization as expected.

The following factors are important during tool selection:

i. Assessment of the organization’s maturity (e.g. readiness for change);

ii. Identification of the areas within the organization where tool support will help

to improve testing processes;

iii. Evaluation of tools against clear requirements and objective criteria;

iv. Proof-of-concept to see whether the product works as desired and meets the

requirements and objectives defined for it;

v. Evaluation of the vendor (training, support and other commercial aspects) or

open-source network of support;

vi. Identifying and planning internal implementation (including coaching and

mentoring for those new to the use of the tool).

V. When to Use Automated Test Tools, Testing Using Automated Tool.

1. When Does Test Automation Make Sense?

(Question: Explain the factors considered when testing makes sense. – 4

Marks) i. When there are many repetitive tests.

ii. When there are frequent regression testing iterations. iii. When you need to simulate large number of users who are using the

application resources. iv. When AUT is having comparatively stable UI.

Software Testing

Anuradha Bhatia 10

v. When you have large set of BVT cases. vi. When you can’t rely solely on manual test execution for critical functionality

Here are the criteria you need to consider before selecting any testing tool: i. Do you have necessary skilled resource to allocate for automation tasks?

ii. What is your budget? iii. Does the tool satisfy your testing needs? Is it suitable for the project

environment and technology you are using? Does it support all tools and objects used in the code? Sometime you may get stuck for small tests due to inabilities of the tool to identify the objects used in the application.

VI. What are Metrics and Measurements: Types of Metrics, Project Metrics, Progress and Productivity Metrics?

In software testing there are three main areas which needs to be considered while

thinking about metrics and measurement.

(Question: Explain the concept of metrics and measurement. – 4 Marks) i. Defining the Metrics

Small and quality set of metrics should be chosen, large set of metrics should be avoided as it is very confusing to understand large set of metrics.

Metrics should also be uniform and everybody in team should agree with it.

ii. Tracking test metrics After defining the metrics the next step is to track the metrics. Since tracking is a constant activity so it’s always nice to automate the

tracking part. Automation reduces time required to track the metrics, analyze them and

measure them. iii. Reporting

Reporting of the metrics is the most important step, you should report test metrics to stakeholders so that they have clear picture of project progress.

A Metric is a quantitative measure of the degree to which a system, system component,

or process possesses a given attribute. Metrics can be defined as “STANDARDS OF MEASUREMENT”. Software Metrics are used to measure the quality of the project. Simply, Metric is a unit used for describing an attribute. Metric is a scale for measurement.

Software Testing

Anuradha Bhatia 11

“How many issues are found in thousand lines of code?”, here No. of issues is one

measurement & No. of lines of code is another measurement. Metric is defined from these two measurements.

1. What is Software Test Measurement?

(Question: Explain the concept of software test measurement. – 4 Marks)

i. Measurement is the quantitative indication of extent, amount, dimension, capacity, or size of some attribute of a product or process.

ii. Test measurement example: Total number of defects. iii. Please refer below diagram for clear understanding of the difference

between Measurement & Metrics.

2. Why Test Metrics?

Generation of Software Test Metrics is the most important responsibility of the Software Test Lead/Manager.

Test Metrics are used to, i. Take the decision for next phase of activities such as, estimate the cost &

schedule of future projects. ii. Understand the kind of improvement required to success the project

iii. Take decision on process or technology to be modified etc. Importance of Software Testing Metrics:

Software Testing

Anuradha Bhatia 12

i. As explained above, Test Metrics are the most important to measure the quality of the software.

ii. Now, how can we measure the quality of the software by using Metrics? iii. Suppose, if a project does not have any metrics, then how the quality of

the work done by a Test analyst will be measured?

For Example: A Test Analyst has to, i. Design the test cases for 5 requirements

ii. Execute the designed test cases iii. Log the defects & need to fail the related test cases

iv. After the defect is resolved, need to re-test the defect & re-execute the corresponding failed test case.

In above scenario, if metrics are not followed, then the work completed by the test analyst will be subjective i.e. the test report will not have the proper information to know the status of his work/project. If Metrics are involved in the project, then the exact status of his/her work with proper numbers/data can be published.

I.e. in the Test report, we can publish: 1. How many test cases have been designed per requirement?

2. How many test cases are yet to design? 3. How many test cases are executed? 4. How many test cases are passed/failed/blocked? 5. How many test cases are not yet executed? 6. How many defects are identified & what is the severity of those defects? 7. How many test cases are failed due to one particular defect? etc. Based on the project needs we can have more metrics than above mentioned list, to know the status of the project in detail.

Based on the above metrics, test lead/manager will get the understanding of the

below mentioned key points. a) %ge of work completed b) %ge of work yet to be completed c) Time to complete the remaining work d) Whether the project is going as per the schedule or lagging? etc. Based on the metrics, if the project is not going to complete as per the schedule, then the manager will raise the alarm to the client and other stake holders by providing the reasons for lagging to avoid the last minute surprises.

Software Testing

Anuradha Bhatia 13

3. Metrics Life Cycle

(Question: Draw the diagram of metrics life cycle. – 4 Marks)

4. Types of Manual Test Metrics:

Testing Metrics are mainly divided into 2 categories. 1. Base Metrics

Base Metrics are the Metrics which are derived from the data gathered by the Test Analyst during the test case development and execution.

This data will be tracked throughout the Test Life cycle. I.e. collecting the

data like, Total no. of test cases developed for a project (or) no. of test cases need to be executed (or) no. of test cases passed/failed/blocked etc.

2. Calculated Metrics Calculated Metrics are derived from the data gathered in Base Metrics.

These Metrics are generally tracked by the test lead/manager for Test Reporting purpose.

Software Testing

Anuradha Bhatia 14

5. Examples of Software Testing Metrics:

Let’s take an example to calculate various test metrics used in software test reports:

Below is the table format for the data retrieved from the test analyst who is actually involved in testing:

6. Importance of metrics and measurement in SDLC

i. During all the software development life cycle it is very important to apply

metrics and measurement because metrics and measurement set

expectations. If there are well established metrics and measurements in

project then the test analyst can easily track and report quality results to

the management.

ii. If the metrics & measurements are not established properly then the

assessment of software quality is purely subjective which arises disputes

at the end of development life cycle. You can consider some the following

Software Testing

Anuradha Bhatia 15

areas where you can apply metrics and measurement. This list is not

exhaustive, you can have metrics for lot more things. Schedule of project Coverage Planned & actual cost Workload & resource usage Product risk & project risk Defects

iii. While doing test planning we set the expectations for the stakeholders or

we set the baselines for them. If you have established the baselines the

test reporting is consistent to the management and you can avoid

subjective assessment of testing.