Post on 31-May-2020
CONTACTEmil Alégroth – Emil.Alegroth@Auqtus.com
WEBwww.auqtus.com
Testing Virtual Reality (And other complex software environments!)
Dr. Emil AlégrothAuqtus | Blekinge Institute of Technology | Chalmers University of Technology
• We delivering state-of-art testing solutions to industry with focus on Visual GUI Testing (VGT).
• Founded by academics and technical experts in the area of automated testing.– Combined; over 50 years of academic experience– 100s of academic publications– World-leading researchers on VGT (Founded the term)
Contributors to the talk:
Dr. Emil Alégroth Prof. Robert Feldt Dr. Simon Poulding
What is Virtual Reality?
• Oculus rift and Google Cardboard• Leisure applications
• Games• Communication
• Learning applications
Our application: VRHangout
Unity
• Game engine
• All in-game fixtures treated as objects– Makes scripting easy
• C# scripting– Quick and powerful
– Same syntax as java
• Many plug-ins– Google cardboard!
• Main limitation– Standard components are generally slow!
Unity game: Hearthstone
My experiences with Unity (5h vs 5 days)
Unity
VR is only the beginning!
Augmented reality
• Leisure applications (e.g. Minecraft)• Life-critical applications
• Medical systems
Safety requirements willrequire stringent qualityassurance!
How do we test in this environment?
• Low-level:– Automated unit tests and integration tests– C# components (in Unity)
• High-level:– Manual testing
• Why?– The real-time, virtual environment, is (often) non-
deterministic and complex to test– Complexity: Computationally heavy to test, difficult to find
oracles, etc.
A complex environment requires complex testing
• Search-based software testing (SBST)– Test case generation, based on a (*)fitness function, that
can automatically determine if the output of the generated test case is (**)correct/faulty/interesting to/for the tester.
• What is a (*)?– Domain model that defines rules for how to, for instance,
generate suitable test cases (A heuristic). Can be modified to learn during execution (Machine learning).
• How do we determine (**) in a VR environment?– For instance: Kolmogorov complexity!
G
Test case generation with Meta-Heuristics
• Meta-heuristic– A description of a possible solution to generate test cases– For instance: Genetic algorithms
T1T2
T2 T1 T2
T1
T2
T1 T2T1Determine best child with a fitnessfunction (best fit)
Gen 1 seed
Gen 1
Gen 2 seed T2T1T1
Oracle?
• Oracle– A means of determining test case outcome– Examples: Value, image, strings or other output
• Kolmogorov complexity allows you to determine a distance between any two objects!
• The measure gives a distance between two objects– Determine if a state is potentially correct/faulty/interesting
Kolmogorov complexity example usage
SBST + Kolmogorov
• Generate exploratory test cases, verify with the automated oracle and learn by testing!
• Small problem…
– Kolmogorov komplexity can NOT be computed!
– Theoretical concept...
• …BUT! It can be approximated with compression!
Compress((1,[1,3,4,5]) , (2,[1,4,5])
) = ([1,4,5],(1,[3]))
Compress( , )
= ( ,(255,0,0),(0,0,255))”Distance”
”Distance”
No tools do this yet butthey are on the way!
VR and AR are tomorrow. We need tools/techniques today!
GUI
GUI model
System architecture
System components /Source code
System model
Multiple languages
Missinginterfaces
Too muchcode
Etc.
Encapsulateddesign
Distributedsystem
Etc.
Customcomponents
Limitedusability
Etc.Maintenance
costs
Speed Robustness Etc.
FAILURES
DEFECTS
CHALLENGES
MORE SUPPORT
STRONGER NEED
(e.g. Continuous
development)
Visual GUI Testing (VGT): JAutomate
• Seamless testing of different applications• End-to-end testing• Advanced testing through the GUI• Non-functional testing (e.g. Performance)• Etc.
VGT: Primarily System- and Acceptance Regression testing
• VGT-GUITAR– Automated Exploratory, model-based, GUI testing
– State space explosion problems– Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing:
an Empirical study
• What if we could provide the tester with a heads-up display to improve exploratory testing based on user domain knowledge?
• What if we could automatically record exploratory test cases for regression testing?
Augmented exploratory testing
Circles: covered statesColor: Likelyhood of finding defect
Expand: Get more information
Fantastic!!! but is it a
Fantasy???
Current state
Neural networks combined with advanced search algorithms
Home
v
Neural net for Version X+1
• Each node has:• A screenshot of the state of the SUT• The interaction to move to the next node
COMING SOON!
If advanced model,automatic exploration!
Conclusions
• VR and AR require more complex test techniques
• Search-based testing with Kolmogorov complexity
– Not available yet!
• Visual GUI Testing
• Augmented Exploratory Testing
Emil.Alegroth@Auqtus.com
Thank you! Questions?
E. Börjesson, R. Feldt, “Automated System testing using Visual GUI Testing Tools: A Comperative Study in Industry”, International Conference on Software Testing, 2012.E.Alégroth, R.Feldt, H. Olsson, “Transitioning Manual System Test Suites to Automated Testing: An Industrial Case Study”, International Conference on Software Testing, 2013.E. Alégroth, M. Nass, H. Olsson, “JAutomate: a Tool for System- and Acceptance-test Automation”, International Conference on Software Testing, 2013. E. Alégroth, R. Feldt, L. Ryrholm, “Visual GUI Testing in Practice: Challenges, problems and limitations”, Empirical Software Engineering Journal, 2014E. Alégroth, Z. Gao, R. A.P. Oliveira, A. Memon, “Conceptualization and Evaluation of Component-based Testing Unified with Visual GUI Testing: An Empirical Study”, International Conference on Software Testing, 2015.E. Alégroth, R. Feldt, P. Kolström, “Maintenance of Automated Testing in Industry: An Empirical study on Visual GUI Testing”, Journal of Systems and Software (In submission).E. Alégroth, J. Gustafsson, H. Ivarsson, R. Feldt, “Replicating Rare Software Failures with Visual GUI Testing: An Industrial Success Story”, IEEE-Software.
Emil.Alegroth@Auqtus.com