Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP...
-
Upload
stag-software-pvt-ltd -
Category
Technology
-
view
720 -
download
0
description
Transcript of Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP...
![Page 1: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/1.jpg)
Hypothesis Based Testing - Application and
Adaptation for testing Enterprise Software Satvik Kini, Suite Test Center, SAP Labs India Pvt Ltd., Bangalore
December 19, 2013 Public
![Page 2: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/2.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 2 Public
Agenda
Hypothesis Based Tests – Overview
Traditional Test Approach vs. HBT
Stages in HBT Approach – (Detailed Explanation with a case study)
Challenges and Benefits
Key factors for success
Summary
Q&A
![Page 3: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/3.jpg)
Hypothesis Based Tests - Overview
![Page 4: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/4.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 4 Public
Hypothesis Based Tests - Overview
Hypothesis :
Tentative explanation for a certain behavior, phenomenon, events that have occurred or will occur
• Null Hypothesis – H0 – Eg: Heavy Smoking is the cause of Lung Cancer
• Alternate Hypothesis – H1 – Eg: Heavy Smoking is not the cause of Lung Cancer
Hypothesis Based test:*
Goal focused approach, test methodology based on solid scientific principles built on the core theme of
hypothesizing potential defects.
* HBT is the IP of STAG Software.
![Page 5: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/5.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 5 Public
Hypothesis Based Tests - Overview
The Scientific Method
Form a Question
Research
Hypothesis
Experiment
Analyze Results
Draw Conclusions
Does my software work ?
Understand Context
Hypothesize Defects
Develop & Execute Tests
Pass or Fail ?
Does my software work ?
![Page 6: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/6.jpg)
Traditional Test Approach Vs. HBT
![Page 7: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/7.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 7 Public
Traditional Test Approach Vs. HBT
Activities
Manual Test Cases
Automated Test Cases
----------------------------
----------------------------
----------------------------
Defects
Goal
Risks/
Potential
Defects
Activities
Test Levels
Test Design
Test Execution
----------------------------
----------------------------
Determines
Drives
Clean
Software
Hopefully results in
Focus: Coverage, Automation,
Test Scripts Focus: Defect Elimination based
approach for cleanliness
Defect Elimination
![Page 8: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/8.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 8 Public
Traditional Test Approach - Limitations
Limitations of Traditional Approach
• Often Software is passed with X% OK Rate
• Assumption – existing test cases cover desired parts of the software
• Hopeful encounter of bugs
• Focus on Quantitative reporting
![Page 9: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/9.jpg)
Stages in HBT
![Page 10: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/10.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 10 Public
HBT – An Example
Example 1 : Clean Water
Goal: Having Clean Water
Potential Defects in water: color, odour, suspended particles, bacteria
Cleaning Process:
• Conduct activities to first determine if each of the potential defects exists
• If a potential defect exists, then take steps to clean that defect
• Once all defects are removed, re-test the water for all potential defects and certify that the water is
clean
Result: Clean Water (goal is achieved)
![Page 11: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/11.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 11 Public
HBT - Stages
Perform Testing Analyze and
Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
(PDT)
Understand End
user expectation,
System Behavior
Consider All
possible different
defect types and
identify each with
test quality levels
Use of different
test techniques
and tools to test
each of the PDTs
Document and
Analyze the
results and get
an overall picture
on Quality
![Page 12: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/12.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 12 Public
Stages in HBT – Case Study
Scenario – Student Course Registration and Completion
Record External
Achievements
Equivalency
Determination Course Registration
Attendance Tracking
Examination Grading Produce Transcript
User Types
• Student
• Admin
• Academic
Advisor/Instructor
Market Places
• US Region
• Europe Region
• APJ Region
![Page 13: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/13.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 13 Public
HBT – Cleanliness Criteria (Goal)
Example 2 : End to End (E2E) Scenario tests
Goal : The E2E Scenario Student Course Registration and Completion works fine
Perform Testing Analyze and
Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
![Page 14: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/14.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 14 Public
HBT – Define Potential Defect Types
Perform Testing Analyze and
Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
![Page 15: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/15.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 15 Public
HBT – Test Levels
Level 5
Expert Level E2E Tests
Level 4
Advanced E2E Scenario
Tests
workflows; role based
tests;Accessibility
Integration across
scenarios
Level 3
Integration tests
Multiple Users;
Timezone based;
Less Used features
Level 2
Primary E2E Use Cases
Integration defects;
Reporting
usability
Level 1
Transactional Coverage
Key E2E Requirements
Browser compatibility
Performance
Obvious Errors
Runtime Errors
Commonly used E2E
Use cases
• Each level corresponds to a set of potential Detect Types (PDTs)
• A purposeful detection of PDTs
• Only once all PDTs are tested for, a level is called to be reached
![Page 16: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/16.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 16 Public
HBT – Test Matrix
Test levels Potential Defect Types
Level 1 Runtime Errors Obvious Functional Errors Commonly used E2E usecases
Level 2 Broswer Compatibility Calculations Performance
Level 3 Integration Reports Usability
Level 4 Currency conversion Less used features Timezone based
Level 5 Workflows Multiple Users Accessibility Integration to other scenarios
Creation of Report Card - Derive Test Matrix
![Page 17: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/17.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 17 Public
HBT – Perform Testing
I. Use of Existing Test Techniques/Tools : 1. Test case driven testing or scripted testing
a. Automated test cases (only scripted tests)
b. Manual test cases (also giving the tester liberty of ad-hoc tests)
2. All pairs testing
3. Exploratory testing
4. Usage of Specific Test tools
II. Preparation of Charters
Perform Testing Analyze and
Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
![Page 18: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/18.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 18 Public
HBT – Analyze and Conclude
• Document the results using charters/protocols
• Check the test results for each of the PDTs
• Update Report Card/Test Matrix - get a overall picture on Product Quality
Perform Testing Analyze and
Conclude
Define
Cleanliness
Criteria
Define
Potential
Defect Types
![Page 19: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/19.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 19 Public
HBT – Test Matrix
Update Report Card
Test levels Potential Defects Types
Level 1 Runtime Errors Obvious Functional
Errors
Commonly used E2E
usecases
Level 2 Broswer Compatibility Calculations Performance
Level 3 Integration Reports Usability
Level 4 Currency conversion Less used features Timezone based
Level 5 Workflows Multiple Users Accessibility Integration to
other scenarios
![Page 20: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/20.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 20 Public
Summary - HBT Realization
Prepare PDT List
Derive Test Matrix
Prepare Charters
Test Using Charters
Update Report Card
Protocol Logging
Define Cleanliness Criteria
![Page 21: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/21.jpg)
Challenges, Benefits and Key Factors for
Success
![Page 22: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/22.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 22 Public
Challenges
Ads
Challenges
• Understanding different user contexts –
market places
• Existence of custom code on top of
standard
• Highly configurable software –
customization and Business Functions
![Page 23: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/23.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 23 Public
Benefits
Ads
• Confidence on what is tested
• Exploration of new scenarios as we do
the tests
• Reduction in customer messages
• Test Matrix gives an overall picture of
quality
• Higher Probability of finding defects as
the goal is to check existence of PDT
• Improvement - Code Coverage
Benefits
![Page 24: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/24.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 24 Public
Key Factors for Success
Ads
Key Factors for Success
• Success mainly depends on
Understanding internal and external
factors and list of PDTs
• Identifying each PDT and its
corresponding level
• For each of the PDT, choosing right set
of scenarios/risks
• Choosing right set of tools and test
techniques to test each PDT
![Page 25: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/25.jpg)
Summary
![Page 26: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/26.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 26 Public
Summary
Goal focused approach, test methodology based on solid scientific principles built
on the core theme of hypothesizing potential defects.
![Page 27: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/27.jpg)
Q&A
![Page 28: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/28.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
Thank you
Contact information:
Satvik Kini
Associate Quality Manager
SAP Labs India Pvt. Ltd, Bangalore
+91 98863 89654
![Page 29: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/29.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved.
APPENDIX
![Page 30: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/30.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 30 Public
Stages in HBT - Realization
Example of Charter :
Scope PDT Risks To Manage How Test Techniques Tester Messages Additional Comments
All Business Processes
Common Used E2E use cases
Check the straight forward E2E scenario is working
Run Automated Test cases
Automated test cases Garbage Collector Tour
Mr. A
Attendance Tracking UI
Browser Compatibility
Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari
Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display
Exploratory Tours : Super Model Tour, Garbage Collector Tour
Mr. B
Equivalency Determination
Obvious Functional Errors
Mapping of External Achievements to Internal subjects
Try different achievements with different status and validity dates
Exploratory Tour : Intellectual Tour All pair Testing
Mr. C
BACK
![Page 31: Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs](https://reader035.fdocuments.us/reader035/viewer/2022062615/5482617cb07959570c8b4786/html5/thumbnails/31.jpg)
© 2013 SAP AG or an SAP affiliate company. All rights reserved. 31 Public
Stages in HBT - Realization
Update each Charter after testing each of the PDTs
Scope PDT Risks To Manage How Test Techniques Tester Messages Additional Comments
All Business Processes
Common Used E2E use cases
Check the straight forward E2E scenario is working
Run Automated Test cases
Automated test cases Landmark Tour
Mr. A None All Business processes are working fine
Attendance Tracking UI
Browser Compatibility
Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari
Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display
Exploratory Tours : Super Model Tour, Garbage Collector
Mr. B 23348743 32476764
Checked browser compatible for the IE9,chrome,Safari . Yet to check for Firefox
Equivalency Determination
Obvious Functional Errors
Mapping of External Achievements to Internal subjects
Try different achievements with different status and validity dates
Exploratory Tour : Intellectual Tour All pair Testing
Mr. C 43234234 Runtime Error while clicking on the Sort Button Blocked
BACK