Webinar - 'Test Case Immunity’- Optimize testing

Webinar: Jan 7, 2016 | 1100-1200IST

T Ashok Founder & CEO, STAG Software Architect - HBT

in.linkedin.com/in/AshokSTAG Ash_Thiru

Welcome to the webinar on “Test Case Immunity – Optimize testing”. In this webinar we will look at an interesting idea ‘Test case immunity” and how can help us in optimizing testing.

© 2016. STAG Software Private Limited. All rights reserved. 2

Let us start with a story…

A poor farmer loses his crop and is advised to use pesticide.

The next season around, he sprays the crop with DDT killing the pests and improving the yield.

A few seasons later, the pests become resistant to DDT and now he is advised to switch to a different pesticide 'Malathion'. The yield improves but the story repeats again after a few seasons.

From the book ‘Software Testing Techniques by Dr Boris Bezier’

The pesticide you use to kill the pest in the system

makes the system resistant to the pest!

THE PESTICIDE PARADOX

We all must be familiar with the story about a poor farmer if we have read the very famous book ‘Software Testing Techniques by Dr. Boris Bezier’. The story is about a poor farmer who loses his crop and is advised to use pesticides. The next season around, he sprays the crop with DDT, killing the pests and improving the yield. A few seasons later, the pests become resistant to the pesticide (DDT), and he is advised to switch to a different pesticide. The yield improves but the story repeats again after a few seasons. The pests become resistant to ‘Malathion’ also. This story indicates an interesting paradox : “The pesticide that we use to kill the pest in the system makes the system resistant to this pest”. This is famously know as – ‘The Pesticide Paradox’. So what is the connection to our webinar? This is where IMMUNITY steps in, where the system has become resistant to the test cases.

THE PESTICIDE PARADOX | IMMUNITY

© 2016. STAG Software Private Limited. All rights reserved.

The ability of the system to resist being affected. A measure of strength, fitness, cleanliness.

i.e. test cases do not uncover any defects.

3

That the pesticide doesn’t work on the system anymore means that the system has become immune to the pesticides. A pest in our context are software defects, pesticides are test cases and the field , the software under test. Therefore immunity is the ability of the system to resist from being affected i.e. the test cases are not finding any issues. It is a measure of strength, fitness, cleanliness of the system. So pesticide paradox and the Immunity are the same. Pesticide Paradox is from the pesticide’s stand point whilst Immunity is from the system stand point.

We execute test cases, record defects & analyse these to ascertain the quality.


Do you analyze test cases that have not yielded any defects over the last few runs?

When we execute test cases, our interest lies in those test cases that have failed and we record the defects and analyze the same to ascertain quality. “Do we analyze the test cases that have not yielded any defects over the last few runs? “Our focus in the webinar is to analyze those test cases that have not yielded any defects over the last few runs to ascertain which parts of the system have become immune or fit and use that information to optimize testing.

Let us analyse the execution data…


Run#1 Run#2 Run#3 Run#4

F1

TC 1 ✔ ✔ ✔ ✔

TC 2 ✔ ✔ ✖ ✔

TC 3 ✖ ✖ ✔ ✔

TC 4 ✖ ✔ ✔ ✔

F2

TC 5 ✔ ✔ ✖

TC 6 ✔ ✔ ✔

TC 7 ✖ ✔

TC 8 ✔ ✔

TC 9 ✔

TC 10 ✔

<== Immune

<== Immune

<== Immune

F ==> Feature TC ==> Test Case

Let’s have a look at the sample data of execution. We have here 10 test cases which cover some part of the features F1 and F2. ✓ indicates pass and ✗ indicates fail. As we can see TC1, TC6 and TC8 are continuously passing. TC1 has been passing Run1-4, TC6 Run2-4 whilst TC8 Run3&4 i.e. no defects reported in all the contiguous runs.. That those areas that are exercised by these test cases, some of which belong to Feature1 and some to Feature2 are immune and therefore those areas are clean.

Visualising a system


E1 E2

E3 E4

See the system as a composition of entities. An entity Ei may be a technical feature, business requirement, user story, end user flow.

A system can be seen as a composition of entities where an entity may be a feature, a business requirement, user story or end user flow. As we execute test cases, we exercise different part of these areas that are covered of the entities E1-E4.

Immunity helps in VISUALISING FITNESS


E1 E2

E3 E4

Test cases of each entity that have ‘passed’ over all the runs indicates areas that are immune.

…that those areas are FIT.

If certain part in the system has become immune i.e. the existing test cases do not being yield any issues, the system will look like as shown here in the “white “areas i.e. these white areas haven’t yielded any defect in the last few runs because the test cases of these entities have passed. This gives us an ability to visualize the fitness. Hence those areas that have become immune are the areas that are FIT and give us the indication of Cleanliness or Fitness.

So how can IMMUNITY help?

8

Reduce rework ... by analysing modifications that make the system UNFIT NON-IMMUNE parts of the system are UNFIT

2

Reduce our work ... by NOT testing those parts of the system that are FIT IMMUNE parts of the system are FIT

1


So how can Immunity help?Reduce our work – by NOT testing those parts of the system that are FIT. If test cases are not yielding any defects as indicated by those areas to be immune, we should refrain from executing these unless it is completely automated. Hence we can reduce our work.Reduce rework – by analyzing modifications which causes system to break. In some runs the test cases did not yield any issues, but after modification they make the system UNFIT i.e defects were found and therefore becoming NON-Immune. So if we see a pattern where certain changes are resulting in certain areas becoming NON-Immune then we can be careful in doing our changes. Therefore Immunity can be useful in reducing our rework.

9

Effort = Work x Time🔽 Effort =Work x 🔽 Time

Do faster - AUTOMATION

We know that

Effort =Work x Time

🔽 Effort = 🔽 Work x Time

Do less - IMMUNITY

How can we OPTIMIZE?


So what do we mean by Optimize? How can we optimize?We all know that Effort = Work x Time. If we want to reduce the effort i.e. optimize the effort, we could reduce either Work or Time. Most often we reduce time by doing faster via automation. So automating test cases is a most commonly used to optimize effort.In the context of immunity we want to focus on how we can reduce our work or rework, i.e. doing less by understanding those areas in a system that have fundamentally became Immune. Some of our test cases are titled as cadence tests which runs very regularly to give us some indication of “heart beat” of the system.

Answer: NO.

10

Cadence


Q: Do cadence tests indicate immunity?

Being ALIVE does NOT mean that the system is FIT.

Immunity is an indication of FITNESS. Cadence is an indication of ALIVENESS.

Cadence tests indicate that the system is alive. That the ‘heart is beating’ well. These test cases are expected to pass every time.

Do cadence tests indicate immunity? The quick answer is NO. A cadence test indicates that the system is alive. That the ‘heart is beating’ well. Therefore these test cases are expected to pass every time.Being alive does not mean that the system is FIT. Similar to the fact that a beating heart does not mean that the body is completely FIT. Therefore cadence is an indication of “ALIVENESS” whereas the Immunity is an indication of “FITNESS”.

11

Cadence


Let us summarise…

Immunity is an indication of FITNESS.Analyze test cases that have not yielded any defects.

Immunity can help to

Reduce rework ... by analysing modifications that make the system UNFIT NON-IMMUNE parts of the system are UNFIT

2

Reduce our work ... by NOT testing those parts of the system that are FIT IMMUNE parts of the system are FIT

1

Summarizing what we discussed : We looked at the test cases that have passed continuously in the last few runs and analyzed these to get an understanding of system immunity which is an indication of fitness. We can use this to:Help us reduce our work by not testing those parts of the system that are FIT i.e. immune.Reduce rework by analyzing those test cases are oscillating between Pass and Fail and therefore they are not immune enabling is us to be more careful of the modifications. This entire idea of immunity came from a scientific approach of thinking that we follow which is Hypothesis Based Testing(HBT)

What is

Hypothesis Based Testing (HBT)?

12

HBT is a personal scientific methodology that enables you to test smartly.

$

@ &

#

~ ^

+ - :

Different PDT

$ L9 Business value

@ L8 Deployent correctness

& L7 Attribute correctness

# L6 Environment correctness

~ L5 Flow correctness

^ L4 Behavior correctness

+ L3 Structural correctness

- L2 Interface correctness

: L1 Data correctness

Quality levelsuncovered at

Run#1

Run#2

Run#3

Run#4

TC 1 ✔ ✔ ✔ ✔

TC 2 ✔ ✔ ✖ ✔

TC 3 ✖ ✖ ✔ ✔

TC 4 ✖ ✔ ✔ ✔

TC 5 ✔ ✔ ✖

TC 6 ✔ ✔ ✔

TC 7 ✖ ✔

TC 8 ✔ ✔

TC 9 ✔

TC 10 ✔

By specific TCsby

Curious about HBT? Goto hbtcentral.org

PDT=Potential Defect Type


What is Hypothesis Based Testing (HBT)? It is a personal scientific methodology that enables us to test smartly. In HBT, the System Under Test (SUT) is evaluated in a goal focused manner by first setting up Cleanliness Criteria that is an indication of expectations of Quality and then focusing on the potential type of defects which matched the criteria. And then coming up with the set of test cases in the layered manner - Nine levels, L1 thru L9 so that we can evaluate and asses the cleanliness. Here is a picturisation of the same. There can be different types of potential defects that a system can have (PDT) and to be able to uncover it efficiently/efficiently in a scientific manner in HBT, we look at test cases organized in various levels, L1 – L9. As we cross a level, we become clean to a certain degree. At L1, it is about ensuring that test case which when executed give us an indication that “handling of basic data is good enough” i.e. system does not except bad data. L2 is about ascertaining if the interfaces that accept these inputs are fit enough.When we take the test cases and arrange them across these levels it gives us a better and deeper sense of fitness because it indicates Immunity not just across the systems but gives us more stratified view as shown in the Quality levels L1-L9 and Run1-4.In the example shown above TC1-10 are evaluating the “Behaviour correctness’ at L4.Hence we can say TC1, TC6 and the TC8 relate to the fitness of functionality of the Features F1 & F2.When different PDT uncovers different quality levels by specific test cases we could get a better sense of fitness(immunity) which can be used to assess, if it is structured in the manner show above. If you are interested to know more about HBT goto www.hbtcentral.org

13© 2016. STAG Software Private Limited. All rights reserved.

IMMUNITY ANALYSER : An utility to analyse immunity1

2

We have used this idea to build a small utility to analyze immunity, titled ”Immunity Analyser”. Firstly (Step#1)we import the test cases, then segregate them into various levels QL1 – QL4 and add the run information about the test case execution, whether they have passed or failed at the end of each run.The next step (Step #2) to identify/assess immunity, which is shown in the rightmost table. The third row states 100% which means, that particular feature is very FIT, where as other values say 61 means that 61% of that particular feature in the system is immune. Remember that the test cases are stratified under various levels and therefore if that particular entity say Postal Composed Mail is all about L4 then this immunity is from the functional behavior stand point. As shown here, there are some test cases shown under L1 whist the bulk is at L4 that focuses on behavior validation (of technical feature).


HBT is the intellectual property of STAG Software Private Limited.

www.stagsoftware.com

@stagsoft

blog.stagsoftware.com

Connect with us...

Thank you!

http://www.stagsoftware.com

Webinar - 'Test Case Immunity’- Optimize testing

Software

Transcript of Webinar - 'Test Case Immunity’- Optimize testing