Quality assurance and testing of quality attributes

SoberITSoftware Business and Engineering Institute

HELSINKI UNIVERSITY OF TECHNOLOGY

Quality assurance of non-functional requirements

Mika MäntyläTKK/SoberIT/SPRG

T-76.5650 Software engineering seminar

Mika Mäntylä 2



Outline

IntroductionQA of performance and usability (short)QA of evolvability (long)How are qualities effected by

code review, pair-programming exploratory testing

Practical approach

Mika Mäntylä 3



QA of …

ReliabilityInteroperability

Performance

Usability

Scalability

Security

Availability

Maintainability

Modifiability

Testability

Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Mika Mäntylä 4



… with

TDDModule Testing

Load testing

Exploratory testing

Bounce checker

Test case based testing

Code review

Load tester

Pair-programming

GUI robot

Unit testing

Functional testing

Acceptance testing

External beta-testers

Usability testing

Static analysis

Iteration Demo

Structural testing

Mika Mäntylä 5



TDDModule Testing

Load testing

Exploratory testing


Code reviewLoad tester

Pair-programming

Unit testing

Functional testing

Acceptance testing

External beta-testersUsability

testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance

Usability Scalability

Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Qualities Methods

Mika Mäntylä 6



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Mika Mäntylä 7



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Mika Mäntylä 8



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Qualities and methods have many to many relationship

Mika Mäntylä 9



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Relationships between qualities and methods are poorly understood

Mika Mäntylä 10



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Method may not be the most important factor affecting a quality

Mika Mäntylä 11



TDDModule Testing

Load testing

Exploratory testing



Pair-programming

Unit testing

Functional testing

Acceptance testing


testing

Static analysis

Iteration Demo

Structural testing

Reliability

Interoperability

Performance


Security

Availability

Maintainability

Modifiability

Testability Safety

Survivability

Confidentiality

Portability

Dependability

Timeliness

Integrity

Fault-tolerance

Specific methods for specific qualities

Mika Mäntylä 12



Summary

Qualities and methods have many to many relationship

Often relationships between qualities and methods are poorly understood

Method may not be the key factor when assuring particular quality Code review’s effect on security

Largely dependent on the reviewer’s instructions

Specific methods exist for some qualities Usability testing Load testing

Mika Mäntylä 13



Outline



Practical approach

Mika Mäntylä 14



QA of performance Requirements analysis

Estimate data volumes and user volume & behavior Peak usage vs. Regular usage Realistic data volumes

Acceptable response time Hardware restrictions

With clustered server you can always buy more servers with mobile phone you cannot

Design & Code Testability of the application with load testing tools Design solutions

Bottlenecks, key algorithms, clustering Testing

Data gathered in RA is crucial Automate and parameterizes scripts Measure Use profiler to identify bottlenecks

Mika Mäntylä 15



Nixon B. A., 2000

Mika Mäntylä 16



QA of Usability

Usability – Quality in Use The capability of the software to be understood learned used

and liked by the user under specified conditions (ISO 9126-1)

Understand context and users Design for Usability

Field observation, interviews, prototypes

Usability reviews, e.g. Heuristic evaluation Cognitive walkthrough with task scenarios

Usability testing, e.g. Thinking aloud protocol in usability lab Co-discovery (Pair usability testing) Surveys and questionnaires

Mika Mäntylä 17



Outline



Practical approach

Mika Mäntylä 18



Quality assurance of evolvability

DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support

Mika Mäntylä 19



Method 1 for evolvability: Design

Design for maintainable software Principles

Low and loose coupling System level, E.g.

No database specific functionality Subsystem level, E.g.

Clear separation of responsibilities Interaction through Interfaces

High cohesion Information hiding

Most probable change scenarios should be evaluated Different software component have, e.g. GUI vs. Encryption

different lifetimes different modification rates

Collaborative design Design review

Mika Mäntylä 20



Method 2 for evolvability: Coding conventions Program layout has great impact to program

comprehension Documentation

Naming of code elements Should reflect their intent Naming standard

Commenting Studies have shown that too much comments are also harmful ! Use when naming does not clearly state the intent

e.g. “Check needed because of bug in Java virtual machine”

Structure – Rules for acceptable structure Explicit metrics

“Do not write methods over 100 lines” Or heuristics

“Make sure all methods perform single explicit task”

Mika Mäntylä 21



Method 3 for evolvability: Code review

Spreads knowledge of the system Increases evolvability inderectly

Distribution of CR issue (performance under functionality)

0.00 %

10.00 %

20.00 %

30.00 %

40.00 %

50.00 %

60.00 %

70.00 %

80.00 %

90.00 %

Evolvability Funtionality FP

Industry case Q

Industry case S

Students

Mika Mäntylä 22



Method 4 for evolvability: Tools Code metrics

E.g. Lines of code, number of coupling, execution paths

Quantify the vague concept of evolvability

Predict future development effort

Also industrial adoption Reverse-engineering

i.e. identify the system's components and their interrelationships, e.g. code -> UML

Analyze whether the design matches the actual source

Metrics Pro’s

Provides objective numbers Does not require in depth

understanding of the system Con’s

Does not say anything about the correctness (quality)

e.g. Does the class have the correct associations

Reverse-engineering Pro’s

Provides higher level view of the source code

Can be compared with the intended design

Con’s Requires in depth

understanding of the system

Mika Mäntylä 23



Method 5 for evolvability: Improve

Terms Refactoring - Controlled way to improve the software’s

structure without changing its observable behavior Re-engineering - Examination and alteration of a subject

system to reconstitute it in a new form and the subsequent implementation of the new form

Re-writing – Recreate the product starting with clean slate All of the above can come handy during software evolution

Refactoring - continuous activity performed by all software developers

Re-engineering - when current system structure is no longer usable

Rewriting When re-engineering is not enough When moving to different new technology/language

Mika Mäntylä 24



Method 6 for evolvability: Knowledge distribution

Further development is harder, if expertise is lost Expertise = solutions and locations to domain problems,

domain knowledge, software architecture, design decisions

Good HRM policy can help maintain knowledge Regardless there is risk of losing the people with

knowledge Retirement, moving to area, dying, changing career etc.

Strategies to keep vital knowledge of the system Documenting Spreading knowledge to several people

Mika Mäntylä 25



Method 7 for evolvability: Environment support

Good unit test harness increases evolvability Increases courage to

improve software structure add new features

Reduces developers test efforts

Other environment support can have similar effect “We don't need unit test as we have excessive number

of testers”

Mika Mäntylä 26



Quality assurance of evolvability

DesignCompany Standards / Coding conventionsCode inspectionsAssessing software evolvability with tools Improving software evolvabilityKnowledge of the system Environment support

Mika Mäntylä 27



Outline



Practical approach

Mika Mäntylä 28



Code review’s effect to qualities

Spreads knowledge of the system Increases evolvability inderectly

Distribution of CR issue (performance under functionality)

0.00 %

10.00 %

20.00 %

30.00 %

40.00 %

50.00 %

60.00 %

70.00 %

80.00 %

90.00 %

Evolvability Funtionality FP

Industry case T

Industry case S

Students

Mika Mäntylä 29



Pair programming’s effect to qualities

Increases understandabilityMakes design better (evolvability)

Mika Mäntylä 30



Exploratory testing’s effect to qualaties

Single study: Exploratory testing (ET) vs Test Case Testing (TCT)

ET found more usability and GUI issues TCT found more performance issues

Type ET TCTET/TCT

(%) Total

Documentation 8 4 200 % 12

GUI 70 49 143 % 119

Inconsistency 5 3 167 % 8

Missing function 98 96 102 % 194

Performance 39 41 95 % 80

Technical defect 54 66 82 % 120

Usability 19 5 380 % 24

Wrong function 263 239 110 % 502

Total 556 503 111 % 1059

Mika Mäntylä 31



Outline



Practical approach

Mika Mäntylä 32



Teams with different goals

Rank effort

Rank statem.

Rank memory

Rank clarity

Rank output

Team1 (effort)

1 4 4 5 3

Team2 (statem.)

2-3 1 2 3 5

Team3 (memory)

5 2 1 4 4

Team4 (clarity)

4 3 3 2 2

Team5 (output)

2-3 5 5 1 1

Mika Mäntylä 33



Practical approach

Functionality Reliability Usability Efficiency Maintainability Portability

Code Review 1 2 3 2 1 1

Test sessions 1 2 1 2 3 3

Load Tests 2 1 3 1 3 3

Multi-platform build 3 2 3 3 2 1

Company specific list of practices

Company specific list of quality attributes, e.g.:• Only developing Windows application get rid of Portability• Developing crypto library remove Usability take in Security

Mika Mäntylä 34



Summary

Many quality attributes and many quality assurance methods exist Lecture series instead of single lecture?

Plenty of research on assuring certain qualities Usability, Security, Evolvability (=Maintainability)

Lack of research on qa methods effects on qualities Effects of pair-programming, exploratory testing to

qualities

Exists specific methods for specific qualities Usability testing

Mika Mäntylä 35



Topics

Consulting plan for a companyHow to test Unit tests effect to quality X

Quality assurance and testing of quality attributes

Documents

Transcript of Quality assurance and testing of quality attributes