Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the...
Transcript of Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the...
![Page 1: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/1.jpg)
Testing Red Hat Enterprise Linux
the Microsoft way
Alexander Todorov
FOSDEM 2018
![Page 2: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/2.jpg)
S
![Page 3: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/3.jpg)
![Page 4: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/4.jpg)
What is pairwise testing
![Page 5: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/5.jpg)
16 – 17 November, SofiaISTACON.ORG
![Page 6: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/6.jpg)
• Wheels– 19” or 21”
• Battery (if you remember)– 60 kWh, 75 kWh, 85 kWh or 100 kWh
• Engine– Single or Dual
• Performance mode– Yes or No
![Page 7: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/7.jpg)
2 x 2 x 2 x 4 == 32 combinations
![Page 8: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/8.jpg)
Wheels Battery Engine Performance mode
19 “ 60 kWh Single Yes
19 “ 75 kWh Single Yes
19 “ 85 kWh Single Yes
19 “ 100 kWh Single Yes
21 “ 60 kWh Dual No
21 “ 75 kWh Dual No
21 “ 85 kWh Dual No
21 “ 100 kWh Dual No
![Page 9: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/9.jpg)
![Page 10: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/10.jpg)
I've pairwise testedRed Hat Enterprise Linux install
during the entire test campaign!
across all product variants!
![Page 11: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/11.jpg)
Installation testing 101
![Page 12: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/12.jpg)
16 – 17 November, SofiaISTACON.ORG
![Page 13: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/13.jpg)
16 – 17 November, SofiaISTACON.ORG
![Page 14: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/14.jpg)
9 different product variants
I consider them platform independent
![Page 15: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/15.jpg)
3 test groups: Tier #1, #2 and #3
![Page 16: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/16.jpg)
6000 test case executions
![Page 17: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/17.jpg)
“Insanity - doing the same thing
over and overand expecting
different results.”Albert Einstein
![Page 18: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/18.jpg)
1) Take all platform dependent tests (pairwise where possible)
![Page 19: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/19.jpg)
2) Pairwise all tests with parameters
![Page 20: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/20.jpg)
storage / iSCSI / No authentication / Network init script storage / iSCSI / CHAP authentication / Network Manager storage / iSCSI / Reverse CHAP authentication / Network
• Authentication type: None, CHAP, reverse CHAP (3)• Networking system: NetworkManager or SysVinit (2)
![Page 21: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/21.jpg)
• 3 x 2 == 6• Pairwise: 3 x 2 == 6• Across all variants: 9 x 3 x 2 == 54• Pairwise across all variants: 9 x 3 == 27
![Page 22: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/22.jpg)
3) Randomize tests without parameters
![Page 23: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/23.jpg)
Partitioning / swap on LVM
• No parameters!• Pairwise can't reduce variant as parameter
– 9 x 1 == 9
• Execute on random product variant each time!
![Page 24: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/24.jpg)
Acceptance criteria
![Page 25: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/25.jpg)
Less test case executions
![Page 26: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/26.jpg)
Don't miss existing bugs *
how does pairwise compare to full test suitewrt defect finding abilities ?
![Page 27: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/27.jpg)
Don't increase product risk *
how many critical defects would I missif I don't execute the full test suite ?
![Page 28: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/28.jpg)
Experiment results
![Page 29: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/29.jpg)
65 % less test case executions !
2119 test cases in pairwise test plan
![Page 30: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/30.jpg)
76 % execution completion rate
previous releases are around 85%
![Page 31: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/31.jpg)
3 x 30 % bug discovery rate
![Page 32: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/32.jpg)
30 % of bugs found by Tier #1
good job, test cases not included in experiment
![Page 33: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/33.jpg)
30 % of bugs found by Pairwise
same were detected by following regular test plan
![Page 34: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/34.jpg)
30 % of bugs found by ET
we don't have test cases for them! Ouch !
![Page 35: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/35.jpg)
Pairwise missed 4 critical bugs
3 were regressions
![Page 36: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/36.jpg)
• #1396949 - After installation with ibft the default route is missing
– gPXE, firmware dependent
• #1421039 - Anaconda fails to get kickstart from nfs on s390x
– Corner case on s390x– IPv6 != IPv4
![Page 37: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/37.jpg)
• #1400844 - Interface binding makes iscsi connection fail
– Waived due to bad infrastructure setup– Waived again b/c ComputeNode doesn't support
Specialized Storage
• #1420300 - Certmonger scriptlet outputs errors to update.log during anaconda upgrade
– tested and not being re-tested
![Page 38: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/38.jpg)
Lessons learned
![Page 39: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/39.jpg)
![Page 40: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/40.jpg)
Perform test review regularly
found hidden parameters in tests
found (sort of) duplicate test cases
![Page 41: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/41.jpg)
Observed optimization patterns
combine or pipeline independent TCs
common set-up for multiple TCs across variants
... and pairwise, pairwise, pairwise
![Page 42: Testing Red Hat Enterprise Linux the Microsoft way · Testing Red Hat Enterprise Linux the Microsoft way Alexander Todorov FOSDEM 2018 . S. What is pairwise testing. ISTACON.ORG 16](https://reader036.fdocuments.us/reader036/viewer/2022081407/6051272219730e03b5785602/html5/thumbnails/42.jpg)
Risk of not detecting regressions
risk is significant in Snapshots phase
due to historical aggregation of results