Post on 01-Nov-2014
description
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of non-functional requirements
Mika MäntyläTKK/SoberIT/SPRG
T-76.5650 Software engineering seminar
Mika Mäntylä 2
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
Mika Mäntylä 3
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of …
ReliabilityInteroperability
Performance
Usability
Scalability
Security
Availability
Maintainability
Modifiability
Testability
Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Mika Mäntylä 4
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
… with
TDDModule Testing
Load testing
Exploratory testing
Bounce checker
Test case based testing
Code review
Load tester
Pair-programming
GUI robot
Unit testing
Functional testing
Acceptance testing
External beta-testers
Usability testing
Static analysis
Iteration Demo
Structural testing
Mika Mäntylä 5
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Qualities Methods
Mika Mäntylä 6
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Mika Mäntylä 7
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Mika Mäntylä 8
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Qualities and methods have many to many relationship
Mika Mäntylä 9
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Relationships between qualities and methods are poorly understood
Mika Mäntylä 10
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Method may not be the most important factor affecting a quality
Mika Mäntylä 11
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
TDDModule Testing
Load testing
Exploratory testing
Test case based testing
Code reviewLoad tester
Pair-programming
Unit testing
Functional testing
Acceptance testing
External beta-testersUsability
testing
Static analysis
Iteration Demo
Structural testing
Reliability
Interoperability
Performance
Usability Scalability
Security
Availability
Maintainability
Modifiability
Testability Safety
Survivability
Confidentiality
Portability
Dependability
Timeliness
Integrity
Fault-tolerance
Specific methods for specific qualities
Mika Mäntylä 12
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Summary
Qualities and methods have many to many relationship
Often relationships between qualities and methods are poorly understood
Method may not be the key factor when assuring particular quality Code review’s effect on security
Largely dependent on the reviewer’s instructions
Specific methods exist for some qualities Usability testing Load testing
Mika Mäntylä 13
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
Mika Mäntylä 14
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of performance Requirements analysis
Estimate data volumes and user volume & behavior Peak usage vs. Regular usage Realistic data volumes
Acceptable response time Hardware restrictions
With clustered server you can always buy more servers with mobile phone you cannot
Design & Code Testability of the application with load testing tools Design solutions
Bottlenecks, key algorithms, clustering Testing
Data gathered in RA is crucial Automate and parameterizes scripts Measure Use profiler to identify bottlenecks
Mika Mäntylä 15
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Nixon B. A., 2000
Mika Mäntylä 16
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
QA of Usability
Usability – Quality in Use The capability of the software to be understood learned used
and liked by the user under specified conditions (ISO 9126-1)
Understand context and users Design for Usability
Field observation, interviews, prototypes
Usability reviews, e.g. Heuristic evaluation Cognitive walkthrough with task scenarios
Usability testing, e.g. Thinking aloud protocol in usability lab Co-discovery (Pair usability testing) Surveys and questionnaires
Mika Mäntylä 17
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
Mika Mäntylä 18
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of evolvability
DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support
Mika Mäntylä 19
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 1 for evolvability: Design
Design for maintainable software Principles
Low and loose coupling System level, E.g.
No database specific functionality Subsystem level, E.g.
Clear separation of responsibilities Interaction through Interfaces
High cohesion Information hiding
Most probable change scenarios should be evaluated Different software component have, e.g. GUI vs. Encryption
different lifetimes different modification rates
Collaborative design Design review
Mika Mäntylä 20
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 2 for evolvability: Coding conventions Program layout has great impact to program
comprehension Documentation
Naming of code elements Should reflect their intent Naming standard
Commenting Studies have shown that too much comments are also harmful ! Use when naming does not clearly state the intent
e.g. “Check needed because of bug in Java virtual machine”
Structure – Rules for acceptable structure Explicit metrics
“Do not write methods over 100 lines” Or heuristics
“Make sure all methods perform single explicit task”
Mika Mäntylä 21
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 3 for evolvability: Code review
Spreads knowledge of the system Increases evolvability inderectly
Distribution of CR issue (performance under functionality)
0.00 %
10.00 %
20.00 %
30.00 %
40.00 %
50.00 %
60.00 %
70.00 %
80.00 %
90.00 %
Evolvability Funtionality FP
Industry case Q
Industry case S
Students
Mika Mäntylä 22
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 4 for evolvability: Tools Code metrics
E.g. Lines of code, number of coupling, execution paths
Quantify the vague concept of evolvability
Predict future development effort
Also industrial adoption Reverse-engineering
i.e. identify the system's components and their interrelationships, e.g. code -> UML
Analyze whether the design matches the actual source
Metrics Pro’s
Provides objective numbers Does not require in depth
understanding of the system Con’s
Does not say anything about the correctness (quality)
e.g. Does the class have the correct associations
Reverse-engineering Pro’s
Provides higher level view of the source code
Can be compared with the intended design
Con’s Requires in depth
understanding of the system
Mika Mäntylä 23
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 5 for evolvability: Improve
Terms Refactoring - Controlled way to improve the software’s
structure without changing its observable behavior Re-engineering - Examination and alteration of a subject
system to reconstitute it in a new form and the subsequent implementation of the new form
Re-writing – Recreate the product starting with clean slate All of the above can come handy during software evolution
Refactoring - continuous activity performed by all software developers
Re-engineering - when current system structure is no longer usable
Rewriting When re-engineering is not enough When moving to different new technology/language
Mika Mäntylä 24
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 6 for evolvability: Knowledge distribution
Further development is harder, if expertise is lost Expertise = solutions and locations to domain problems,
domain knowledge, software architecture, design decisions
Good HRM policy can help maintain knowledge Regardless there is risk of losing the people with
knowledge Retirement, moving to area, dying, changing career etc.
Strategies to keep vital knowledge of the system Documenting Spreading knowledge to several people
Mika Mäntylä 25
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Method 7 for evolvability: Environment support
Good unit test harness increases evolvability Increases courage to
improve software structure add new features
Reduces developers test efforts
Other environment support can have similar effect “We don't need unit test as we have excessive number
of testers”
Mika Mäntylä 26
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Quality assurance of evolvability
DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support
Mika Mäntylä 27
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
Mika Mäntylä 28
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Code review’s effect to qualities
Spreads knowledge of the system Increases evolvability inderectly
Distribution of CR issue (performance under functionality)
0.00 %
10.00 %
20.00 %
30.00 %
40.00 %
50.00 %
60.00 %
70.00 %
80.00 %
90.00 %
Evolvability Funtionality FP
Industry case T
Industry case S
Students
Mika Mäntylä 29
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Pair programming’s effect to qualities
Increases understandabilityMakes design better (evolvability)
Mika Mäntylä 30
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Exploratory testing’s effect to qualaties
Single study: Exploratory testing (ET) vs Test Case Testing (TCT)
ET found more usability and GUI issues TCT found more performance issues
Type ET TCTET/TCT
(%) Total
Documentation 8 4 200 % 12
GUI 70 49 143 % 119
Inconsistency 5 3 167 % 8
Missing function 98 96 102 % 194
Performance 39 41 95 % 80
Technical defect 54 66 82 % 120
Usability 19 5 380 % 24
Wrong function 263 239 110 % 502
Total 556 503 111 % 1059
Mika Mäntylä 31
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Outline
IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by
code review, pair-programming exploratory testing
Practical approach
Mika Mäntylä 32
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Teams with different goals
Rank effort
Rank statem.
Rank memory
Rank clarity
Rank output
Team1 (effort)
1 4 4 5 3
Team2 (statem.)
2-3 1 2 3 5
Team3 (memory)
5 2 1 4 4
Team4 (clarity)
4 3 3 2 2
Team5 (output)
2-3 5 5 1 1
Mika Mäntylä 33
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Practical approach
Functionality Reliability Usability Efficiency Maintainability Portability
Code Review 1 2 3 2 1 1
Test sessions 1 2 1 2 3 3
Load Tests 2 1 3 1 3 3
Multi-platform build 3 2 3 3 2 1
Company specific list of practices
Company specific list of quality attributes, e.g.:• Only developing Windows application get rid of Portability• Developing crypto library remove Usability take in Security
Mika Mäntylä 34
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Summary
Many quality attributes and many quality assurance methods exist Lecture series instead of single lecture?
Plenty of research on assuring certain qualities Usability, Security, Evolvability (=Maintainability)
Lack of research on qa methods effects on qualities Effects of pair-programming, exploratory testing to
qualities
Exists specific methods for specific qualities Usability testing
Mika Mäntylä 35
SoberITSoftware Business and Engineering Institute
HELSINKI UNIVERSITY OF TECHNOLOGY
Topics
Consulting plan for a companyHow to test Unit tests effect to quality X