Seminar Software Quality and Safety
description
Transcript of Seminar Software Quality and Safety
![Page 1: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/1.jpg)
1/28
Seminar Software Quality and Safety
Topic: “Comparing design models resultant from two processes: with and without Architecture – an empirical study“
Author: Lukas Banach
Tutors: Nazim MadhavjiRemo Ferrari
Thursday 17.07.2008
Motivation Task Context Variables Instrument Results
![Page 2: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/2.jpg)
2/28
Overview
• Motivation • Task description• Study context• Static / dynamic attributes• Instrument to measure attributes• Preliminary results• Future Work
Motivation Task Context Variables Instrument Results
![Page 3: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/3.jpg)
3/28
Motivation
• Someone wants to create lower-level models• Does architecture help?
• Do we produce higher quality (faster / more precise, ...) with architecture?• What are the costs and the benefits?
• There is a huge gap between research and the industry
• Useful to say: use architecture because of ...
Motivation Task Context Variables Instrument Results
![Page 4: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/4.jpg)
4/28
Motivation
• Answer research question:
“How do lower-level design models constructed with a software architecture compare to lower-level design models that are not based on an architecture?“
Motivation Task Context Variables Instrument Results
![Page 5: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/5.jpg)
5/28
My task description
• My focus is to establish the methology of study to answer this question
• Dependent variables, instument, statistical analysis• Prepare an empirical study• No study – beyond the scope of this seminar
Motivation Task Context Variables Instrument Results
![Page 6: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/6.jpg)
6/28
Basic idea of an empirical study
Independent variable (1)
Dependent variable (292)Dependent variable (38)Dependent variable (35)Dependent variable (436)Dependent variable (007)
Independent variable (0)
Dependent variable (112)Dependent variable (82)Dependent variable (2)Dependent variable (93)Dependent variable (4)
Binary variable (0/1)
influence
Motivation Task Context Variables Instrument Results
![Page 7: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/7.jpg)
7/28
Where are the models from?
• Participants: ungraduate students• Randomly assigned by the staff
• Their task: model the behaviour of a garage door
• Two types of groups• With and without architecture
Motivation Task Context Variables Instrument Results
![Page 8: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/8.jpg)
8/28
Where are the models from?
• 6 groups ouf of 19• XML data (Models)• Task description• All created documents
Motivation Task Context Variables Instrument Results
![Page 9: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/9.jpg)
9/28
Uppaal design models
• Templates• States• Transitions • Xml – Files
• Simulator• Verifier
Motivation Task Context Variables Instrument Results
![Page 10: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/10.jpg)
10/28
Study variables
• Independent variable (binary)• Architecture or no architecture
• Dependent variables• Static attributes• Dynamic attributes
Motivation Task Context Variables Instrument Results
![Page 11: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/11.jpg)
11/28
Static attributes
• Number of templates• Number of states • Number of transitions• Size of the xml file • Number of available features• Detail level of the available features• Complexity Level of the templates
Motivation Task Context Variables Instrument Results
![Page 12: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/12.jpg)
12/28
Dynamic attributes
• Percentage of accessible states • Percentage of correct scenarios• Deadlocks • Infinite loops
Motivation Task Context Variables Instrument Results
![Page 13: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/13.jpg)
13/28
Instrument to measure attributes
• Xml parser• Counting states, transitions, templates & complexity level• Creates queries to check accessible states (E <>)
• Checklist• available features and detail level
Motivation Task Context Variables Instrument Results
![Page 14: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/14.jpg)
14/28
Instrument to measure attributes - Xml parser
Motivation Task Context Variables Instrument Results
![Page 15: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/15.jpg)
15/28
Instrument to measure attributes
• Verifier• Accessible states (E <>)• Deadlocks (E<> deadlock)
• Scenarios• Available features
• Simulator
Motivation Task Context Variables Instrument Results
![Page 16: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/16.jpg)
16/28
Instrument to measure attributes - verifier
Motivation Task Context Variables Instrument Results
![Page 17: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/17.jpg)
17/28
Instrument to measure attributes - simulator
Motivation Task Context Variables Instrument Results
![Page 18: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/18.jpg)
18/28
Collected dataArchitecture teams No architecture teams
Team1 Team3 Team4 Team11 Team12 Team13
Size 73 KB 19 KB 60 KB 55 KB 39 KB 123 KB
Templates 18 9 16 15 28 23
States 167 46 75 59 72 226
Transitions 317 80 155 155 152 412
Complexity Level 2 (states or transitions >=10) 8 2 6 4 4 12
Complexity Level 1 (states and transitions <10) 10 7 10 11 24 11
E<> Queries (out of Memory) 53/54(0) 1/35(0) 76/76(0) 36/53(17) 35/35(0) 48/48(78)
E<> deadlock no yes o.o.m yes o.o.m o.o.m
Motivation Task Context Variables Instrument Results
![Page 19: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/19.jpg)
19/28
Collected data - features
Architecture teams No architecture teams
Feature Team1 Team3 Team4 Team11 Team12 Team13
1a yes (1) no yes (1) yes (1) yes (1) yes (1)
1b yes (1) no yes (1) yes (1) yes (1) yes (1)
1c yes (1) no yes (1) yes (1) yes (1) yes (1)
1d yes (only with key) no yes (2) yes (1) no no
1e yes (1) no yes (1) yes (1) yes (1) yes (1)
1a: Opening the garage door by user pressing the button.
1e: All operations should involve displaying messages on the control unit which would be located in the garage.
Motivation Task Context Variables Instrument Results
![Page 20: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/20.jpg)
20/28
Collected data - scenarios
1: User presses the garage door button to open the garage. While the door is opening, the user presses the button. The user presses the button again; however, there is an obstacle in the way of the door going down.
Architecture teams No architecture teams
Scenario Team1 Team3 Team4 Team11 Team12 Team13
1 yes no yes no yes yes
2 yes no no no no yes
3 no no no no no no
4 no no no no no no
5 no no yes no no no
Motivation Task Context Variables Instrument Results
![Page 21: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/21.jpg)
21/28
Statistical analysis - total
Total
0,00
100,00
200,00
300,00
400,00
500,00
600,00
Size Templates States Transitions
Architecture teams
No architecture teams
Motivation Task Context Variables Instrument Results
![Page 22: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/22.jpg)
22/28
Statistical analysis - percentage
Percentage
0
10
20
30
40
50
60
Architecture teams No architecture teams
Correct features in %
Correct scenarios in %
Motivation Task Context Variables Instrument Results
![Page 23: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/23.jpg)
23/28
Statistical analysis - complexity
Complexity of architecture groups
Complexity Level 2(states ortransitions >=10)
Complexity Level 1(states andtransitions <10)
Complexity of no architecture groups
Complexity Level 2(states ortransitions >=10)
Complexity Level 1(states andtransitions <10)
Motivation Task Context Variables Instrument Results
![Page 24: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/24.jpg)
24/28
Statistical analysis - correlation
-0,4
-0,2
0
0,2
0,4
0,6
0,8
1
Correlation(#Transition -> Features)
Correlation(#States ->Features)
Correlation(#Transition -> Scenarios)
Correlation(#States ->Scenarios)
Architecture
No architecture
Motivation Task Context Variables Instrument Results
![Page 25: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/25.jpg)
25/28
Statistical analysis - significance
• χ2-Test (Chi-Square-Test)
• H1: The correlation between the attribute “Architecture“ and the number of correct Scenarios is statistically significant.
• H0: The attribute “Architecture“ and the number of correct Scenarios are statistically not correlated• χ2 = 0.1893
Motivation Task Context Variables Instrument Results
![Page 26: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/26.jpg)
26/28
Summary
• Attributes
• Created instrument
• Collect preliminary data
• Statistical analysis
Motivation Task Context Variables Instrument Results
![Page 27: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/27.jpg)
27/28
Future work
• Improve scenarios (yes/no -> stopped at )• Create more queries with the xml parser• For the study:
• Use professional people instead of students
Motivation Task Context Variables Instrument Results
![Page 28: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/28.jpg)
28/28
The end
Questions ?
![Page 29: Seminar Software Quality and Safety](https://reader033.fdocuments.us/reader033/viewer/2022051517/568157df550346895dc55df4/html5/thumbnails/29.jpg)
29/28
χ2
A nA
S
3.5 4 3.5 3 7
0.116 0.133 0.116 0.1 0.233
nS
11.5 11 11.5 12 23
0.383 0.366 0.383 0.4 0.766
15 15 30
χ2 = (3.5 – 4)² / 4 + (3.5 – 3)² / 3 + (11.5 – 11)² / 11 + (11.5 – 12)² / 12χ2 = 0.1893
χ2 df = 1:5% = 3.84χ2 df = 1:1% = 6.68χ2 df = 1:0.1% = 10.8