1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa...
-
Upload
roger-richard -
Category
Documents
-
view
214 -
download
0
Transcript of 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa...
![Page 1: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/1.jpg)
1
Common Patterns in Combinatorial Models
Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
IBM Haifa Research Labs
![Page 2: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/2.jpg)
2
Agenda
• Introduction & Motivation
• Pitfalls in Combinatorial Models
• Modeling Patterns
![Page 3: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/3.jpg)
3
Introduction
• Combinatorial model = attributes, values, restrictions
• Correctly identifying them is of the main obstacles in practicing combinatorial testing
• There are common pitfalls, and common patterns that address them
![Page 4: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/4.jpg)
4
Motivation
• Construct a “checklist” for practitioners– Common pitfalls and common solutions
• Raise the topic for community discussion– This is not a complete list– More patterns are welcome
![Page 5: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/5.jpg)
5
Common Pitfalls in Combinatorial Modeling
Correctness
Failing to capture the intention correctly
Completeness
Omitting an important part of the test space from the model
Redundancy
Explicitly enumerating different cases that are actually equivalent
![Page 6: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/6.jpg)
6
Modeling Patterns
• Recurring patterns we encountered in combinatorial models– i.e., recurring properties of the modeled test
spaces
• Address the pitfalls
• Encountered in many different models, regardless of the domain, current level of testing, etc.
![Page 7: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/7.jpg)
7
Overview
![Page 8: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/8.jpg)
8
Optional and Conditionally-Excluded Values (completeness, correctness)
• Example I: online form with fields email (mandatory) and home address (optional)
• Naïve model:– Email – valid / invalid– HomeAddr – valid / invalid
• Much better:– Email – valid / invalid– HomeAddr – valid / empty / invalid
Incomplete !
Does not distinguish between empty and
invalid home address
![Page 9: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/9.jpg)
9
Optional and Conditionally-Excluded Values – cont.• Example II: online form with email
address, home address, and “is billing by email”
• Home address used only if not billing by email
![Page 10: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/10.jpg)
10
• Naïve model: – Email – valid / invalid– HomeAddress – valid / invalid– BillByEmail – true / false
– Not allowed:• HomeAddress == “valid” && BillByEmail• HomeAddress == “invalid” && BillByEmail
Incorrect !
Billing by email was entirely excluded
Optional and Conditionally-Excluded Values – cont.
![Page 11: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/11.jpg)
11
Optional and Conditionally-Excluded Values – cont.
• Much Better:– Email – valid / invalid– HomeAddr – valid / invalid / empty / NA– BillByEmail – true / false
– Not allowed:• HomeAddr != “NA” && BillByEmail• HomeAddr == “NA” && ! BillByEmail
![Page 12: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/12.jpg)
12
Multi-Selection (correctness, completeness)
• Example: shopping cart system. The shopping cart may contain meat, dairy, fish and drinks
• Naïve model:– Cart Item – meat / dairy / fish / drinks
Incomplete and Incorrect !
Does not allow for the cart to contain more
than one item, and does not test interactions
between items
![Page 13: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/13.jpg)
13
Multi-Selection – cont.
• A better solution:– Item 1 – meat / dairy / fish / drinks
…– Item t – meat / dairy / fish / drinks / none
• An even better solution:– Meat – true / false– Dairy – true / false– Fish – true / false– Drinks – true / false
![Page 14: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/14.jpg)
14
Ranges and Boundaries (redundancy, completeness)
• Applied when values of a parameter can be divided into ranges, such that values in a range are equivalent with respect to the testing scenario
![Page 15: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/15.jpg)
15
Ranges and Boundaries – cont.
• Example I: file processing. File of up to 1 MB is handled by a first approach. File of 1-256 MB by a second, and over 256 by a third
• Model:– FileSize: 0 / moreThan0LessThan1 /
1 / moreThan1LessThan256 / 256 / moreThan256
![Page 16: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/16.jpg)
16
Ranges and Boundaries – cont.
• Example II: user updates insurance plan. User chooses benefits for the new plan, some may already exist.
• Model:– Benefits – allBenefitsExisting /
someExistingSomeNew / allBenefitsNew
![Page 17: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/17.jpg)
17
Order and Padding (completeness)
• When? When a requirement defines an output that depends on two (or more) inputs.
• Modeler Q: Can bugs surface if inputs are presented in different orders ?
![Page 18: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/18.jpg)
18
Order and Padding – cont.
• Example: expense reimbursement. Expenses covered only if both food and lodging expenses are submitted
• Naïve model:– Lodging – true / false– Food – true / false
Incomplete !
The following bug may not be detected
![Page 19: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/19.jpg)
19
Order and Padding – cont.
if (record.type == “Food” && pLodgingWasSubmitted){ process(record); }
if (record.type == “Lodgin” && pFoodWasSubmitted){ process(record); }
![Page 20: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/20.jpg)
20
Order and Padding – cont.
• Better model:– Lodging – true / false– Food – true / false– FoodBeforeLodging – true / false /
NotApplicable
![Page 21: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/21.jpg)
21
Order and Padding – cont.
• Modeler Q II: may the implementation depend on the records being consecutive?
• Modeler Q III: may the implementation depend on either records being first / last?
![Page 22: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/22.jpg)
22
Order and Padding – cont.
• Updated model:– Lodging – true / false– Food – true / false– FoodBeforeLodging – true / false /
NotApplicable– PaddingBefore – true / false / NotApplicable– PaddingBetween – true / false / NotApplicable– PaddingAfter – true / false / NotApplicable
![Page 23: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/23.jpg)
23
Multiplicity and Symmetry (redundancy)
• Applicable when:– Multiple elements of the same type– Their interactions with other elements are
equivalent
• Example: two hosts (x86 / PowerPC), two storage devices (Enterprise / Mid-Range / Low-End)
![Page 24: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/24.jpg)
24
Multiplicity and Symmetry – cont.
• Naïve Model:– Host1 – x86 / PowerPC– Host2 – x86 / PowerPC– Storage1 – Enterprise / MidRange / LowEnd– Storage2 – Enterprise / MidRange / LowEnd
Warning: Possible Redundancy !
For example, (host1=x86, Storage1=LowEnd)
may be equivalent to (host2=x86, Storage1=LowEnd)Warning: Possible Redundancy ! For example, (host1=x86, host2=PowerPC)
may be equivalent to (host2=x86, host1=x86)
![Page 25: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/25.jpg)
25
Multiplicity and Symmetry – cont.
• Possible Solution:– numX86Running – 0 / 1 / 2– numPowerPCRunning – 0 / 1 / 2– numEnterpriseRunning – 0 / 1 / 2– numMidRangeRunning – 0 / 1 / 2– numLowEndRunning – 0 / 1 / 2– Not allowed:
• numX86Running + numPowerPCRunning != 2• numEnterpriseRunning + numMidRangeRunning +
numLowEndRunning != 2
![Page 26: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/26.jpg)
26
Multiplicity and Symmetry – cont.
• An alternative approach:
• A tool and algorithm that supports designating equivalent parameters– And whether internal interactions are
equivalent
• Requires a tool that supports dynamically changing coverage requirements.
![Page 27: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/27.jpg)
27
Summary
• Modeling is typically the most challenging part in applying combinatorial testing
• We listed some pitfalls, and common patterns that address them
• There are probably many more
![Page 28: 1 Common Patterns in Combinatorial Models Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick IBM Haifa Research Labs.](https://reader036.fdocuments.us/reader036/viewer/2022062519/5697c0021a28abf838cc2dbc/html5/thumbnails/28.jpg)
28
Thank You For Listening