1 ECOOP 2007 Highlights A Personal Summary Yishai Feldman IBM Haifa Research Lab.
IBM Haifa Research Laboratory
Transcript of IBM Haifa Research Laboratory
© 2012 IBM Corporation��
How can we thoroughly test complex systems and avoid errors of omission?
� Testing is challenging at every level
– Exhaustive testing of complex systems is impossible: many billions of potential tests
– Thorough testing of individual features is difficult: for example, even search functions have multiple options that may have complex interactions
� When people create tests by hand, they tend to fall into a pattern that produces similar tests, most of which add little value in discovering defects. Manually-built test plans tend to have many gaps.
� Combinatorial Test Design (CTD) is a smarter approach that helps us identify the tests that are likely to expose defects. CTD takes a systematic approach to modeling the things that need to be tested, then uses advanced mathematics to dramatically reduce the number of test cases while ensuring coverage of conditions and interactions.
�
End-to-end and system-level testing
Feature-level testing
© 2012 IBM Corporation���
� Source http://csrc.nist.gov/groups/SNS/acts/ftfi.html
The root cause analysis of many bugs shows they depend on a value of one variable (20%-68%)
Most defects can be discovered in tests of the interactions between the values of two variables (65-97%)
Do we really need to test all combinations?
© 2012 IBM Corporation�
Combinatorial Test Design (CTD)
� To balance cost and risk, we select a subset of tests that covers all the interactions of variables at some level of interaction (pairs, three-way, etc.)
� A combinatorial test design (CTD) algorithm finds a small test plan that covers 100% of a given interaction level
© 2012 IBM Corporation�
Combinatorial Test Design is a technique that takes all possible points of variation and generates a small subset of tests based on levels of interaction.
Model the:
Input space
Environment
Scenarios
Generate test set
Generate test configurations
Generate
Combinatorial
Design
Attributes and Values
Test Descriptions
Test set
Test configurations
© 2012 IBM Corporation�
Combinatorial test design is an approach that can be applied to different phases and scopes of testing
End-to-end and system-level testing
Feature-, service- and API-level testing
CTD models at this level typically include
Business transactionsBusiness data and stateSystem implementation
Testing scenarios
CTD models at this level typically
includeInput selections
Input dataNavigation
Error conditions
© 2012 IBM Corporation�
Existing Tests
IBM FoCuS/ CTD solution
The CTD model can be used to optimize tests in multiple ways, taking existing tests and data into account.
CTD model
Find gaps, generate tests to
close
Generate complete set of
tests
Remove overlapping tests Test selection
Test enhancement
Test generation
Test data Select data subset
Distilled test data
Existing Tests
Subset of existing Tests
New Tests to close gaps
New Tests for complete coverage
Test data selection