State-based Robustness Testing of Video Conferencing Systems
Transcript of State-based Robustness Testing of Video Conferencing Systems
![Page 1: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/1.jpg)
Industry-Driven Testing: Past, Present, and Future Activities at Simula
Shaukat Ali
Research Scientist
Certus Software V & V Center
Simula Research Lab
![Page 2: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/2.jpg)
Myself
Affiliated with Simula since 2007
Have been working in collaboration with Cisco for several years
Exploring new testing topics in FMC technologies
Software Quality Engineering Laboratory (SQUALL) , Carleton University, Ottawa, Canada
Siemens Corporate Research (SCR), Princeton, New Jersey, USA.
Verification and Testing Lab (VT), The University of Sheffield, UK
Center for Software Dependability (CSD), Islamabad, Pakistan
![Page 3: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/3.jpg)
Embedded Systems Shape Our World
Home Appliances
Consumer Electronics
Medical Equipment
Transportation
Oil Industry Industrial Robots
![Page 4: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/4.jpg)
Interesting Facts About Embedded Systems
About 98% of computing devices are embedded1
By 2020: 40 billion embedded devices1
Annual budget for embedded systems: 160 billion euros
with annual growth of 9%2
In 2008: 30 embedded microprocessors per person with at
least 5 million function points of embedded software2,3
The growth rate of embedded software is increasing2,3
On average an embedded system has 1 million Lines Of Code
(LOC)
A premium-class automobile has around 100 million LOC
Boeing 787: 6.5 million LOC
1. ARTEMIS, Artemis Joint Undertaking - The public private partnership for R & D Embedded Systems, http://artemis-ju.eu/embedded_systems, 2011
2. Christof Ebert and Capers Jones, Embedded Software: Facts, Figures, and Future, IEEE Computer 42(4), 2009
3. Robert N. Charette, This Car Runs on Code, IEEE Spectrum, 2009
![Page 5: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/5.jpg)
Embedded Systems Can be Critical
Non-critical
Mars Climate Orbiter Mariner I Ariane 5
Mission-critical
Therac-25 Oil rig mishap
Safety-critical
![Page 6: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/6.jpg)
Large-Scale Embedded Systems
Embedded Software in A380
Data Acquisition System in Oil Refineries Communication Intensive
Video Conferencing Systems
Cruise Ships
![Page 7: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/7.jpg)
Certus Software V&V Center at Simula
Industrial Partners
![Page 8: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/8.jpg)
Methodology for Industry-Driven Research at Certus
Adopted from: Tony Gorschek, Claes Wohlin, Per Garre, Stig Lasson, A model for technology transfer in practice, IEEE Software, 2006
![Page 9: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/9.jpg)
![Page 10: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/10.jpg)
Automated Model-based Testing of State-driven Systems (AMOS)
![Page 11: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/11.jpg)
D-Mint Partners
![Page 12: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/12.jpg)
Model-based Test Case Generation: A Simplistic View
UML State machines UML Class diagram
TRUST
//statements to verify state invariant for Idle state testtarget.dial(5345,SIP,2K) //statements to verify state invariant for Connected_1 state
Test Case Generation Tool
Test Scripts
![Page 13: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/13.jpg)
Why Model-based Testing
Facilitates automation
Helps defining specialized test strategies targeting specific test objectives
Systematic and rigorous
Automatically generate many non-repetitive and useful tests
Easier test case maintenance
![Page 14: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/14.jpg)
TRUST supports configurable and extensible features
Input models: State machine, Sequence diagram, ..
Test models: Test tree, ..
Coverage criteria: All transition coverage, All message coverage, ..
Test data generation strategies: Random, Search-based, ..
Test script languages: Python, C++
![Page 15: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/15.jpg)
Model-based Robustness Testing -Cisco
Robustness is the degree to which a software component functions correctly in the presence of exceptional inputs or stressful environmental conditions (IEEE Std 610.12-1990)
![Page 16: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/16.jpg)
Why is robustness important?
Robustness is considered very critical for embedded systems, for example communication and control systems
Required by many quality standards
IEEE Standard Dictionary of Measures of the Software Aspects of Dependability
ISO’s Software Quality Characteristics standard
Software Assurance Standard by NASA
16
![Page 17: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/17.jpg)
MBRT is challenging due to the following reasons:
Modeling robustness behavior makes modeling highly complex and redundant
Automated generation of executable test cases from robustness models
Targeted to reveal robustness faults
Defining appropriate test strategies for robustness testing
Generating test data
17
![Page 18: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/18.jpg)
Case Study: Video Conferencing System
18
![Page 19: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/19.jpg)
How does a Video Conferencing System work in practice?
19
![Page 20: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/20.jpg)
Core Functionality
Saturn
EP1 EP3
Call
Outgoing channel
Incoming channel
Presentation Channel Video
Channel
Audio Channel
20 EP2
![Page 21: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/21.jpg)
Robustness Testing for Video Conferencing Systems
System Under Test
Endpoint 1
Endpoint 2
Endpoint 3
Network X X
X
X
X X
X X
Video
Audio
Presentation
21
![Page 22: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/22.jpg)
High Level Picture of Our Solution
Key results: 95% modeling effort reduced, improved readability, improved
modeling quality, and reduced modeling errors, test data generation
significantly better than existing tools
![Page 23: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/23.jpg)
Test Case Selection
System testing on real platform and network is expensive and time consuming
MBT techniques will generate large sets of test cases when applied to Industrial systems especially in case of non-functional testing
Selecting a small enough subset of these test cases that is realistic to be executed and analyzed during the time and resource constraints, while preserving maximum possible fault revealing power.
![Page 24: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/24.jpg)
Similarity-based Test Case Selection
Goal
Finding a subset of the original test suite, with a given affordable size (based on testing budget), that detects the most faults.
Procedure
Defining a measure for similarity between test cases
Encoding of test cases using right level of abstraction that contains only relevant information
Defining a similarity function that assigns a similarity value per each pair of encoded test cases
Minimizing similarity among selected test cases
Key Result: by selecting 15% of the generated test suite we achieve
almost 100% fault detection rate No need for running 85% of the test
cases.
![Page 25: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/25.jpg)
VERDE: Verification-Oriented & Component-based Model Driven Engineering For Real-
Time Embedded Systems
![Page 26: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/26.jpg)
Overall Testing Framework
Key Results: Comprehensive environment modeling approach for testing,
Automated environment simulator generation, and Automated test case generation
for black-box system testing
![Page 27: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/27.jpg)
![Page 28: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/28.jpg)
Management of product variability in test configuration generation and execution
![Page 29: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/29.jpg)
Management of product variability in test configuration generation and execution
![Page 30: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/30.jpg)
Model-Based functional and non-functional testing of Product Lines
![Page 31: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/31.jpg)
Model-Based functional and non-functional testing of Product Lines
Product Line Engineering (PLE) is expected:
To enhance quality and productivity
Speed up time-to-market
Decrease development effort, through reuse
To use PLE for systematic testing and more specifically model-based testing (MBT)
The aim is to reduce the effort required for modeling and configuring products of the product line family for MBT.
![Page 32: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/32.jpg)
Testing Data-Intensive Systems
![Page 33: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/33.jpg)
Testing Data-Intensive Systems
System and acceptance testing using large data samples
Running of the whole system (as a black box) against test data or a complete simulation of the actual running system for purposes of testing out the adequacy of the system.
Simulation, duplication or replication of parts of a system (whether being a sub-system, super-system or environment) is difficult
Sample selection is of outmost importance
![Page 34: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/34.jpg)
![Page 35: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/35.jpg)
Projects
Testing in real-world configurations and scenarios
Testing by emulating real world configurations such as different Video Conferencing Systems geographically distributed with diverse configurations and with different network configurations as deployed at customers’ sites.
Smart Software Development Solution
Integrated model-based development, code generation, and testing solutions
![Page 36: State-based Robustness Testing of Video Conferencing Systems](https://reader031.fdocuments.us/reader031/viewer/2022022204/6212eddb6fd52b12a37c0ca3/html5/thumbnails/36.jpg)
Questions