T-76.613 Software Testing and Quality Assurance · Runeson and Isacsson, 1998, Software Quality...
Transcript of T-76.613 Software Testing and Quality Assurance · Runeson and Isacsson, 1998, Software Quality...
HELSINKI UNIVERSITY OF TECHNOLOGY
T-76.3601 Basics of Software Engineering
11.4.2008
Software Quality Assurance (SQA)
Juha ItkonenSoberIT
2Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Contents
Software Quality Assurance
Introduction to quality and quality assurance
SQA in CMM(I)
Agile Viewpoint to SQA
+ ( A Bug’s Life )
4Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Definition of software quality
Quality: (1) The degree to which a system, component or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations.
(IEEE Standard Glossary of Software Engineering Terminology [IEEE610.12])
Quality is value to some person(s)(Gerald M. Weinberg, 1992, “Quality Software Management”)
5Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Good Enough Quality
Defined by James Bach IEEE Computer, 1997, vol. 30, no. 8, pp. 96-98.
To claim that any given thing is good enough is to agree with all of the following propositions:
It has sufficient benefitsIt has no critical problemsThe benefits sufficiently outweigh the problemsIn the present situation, and all things considered, further improvement would be more harmful than helpful
Aiming to Good Enough Quality does not necessarily allow us to do less quality practices
We must find the critical problems and at least be able to estimate the risks of the remaining (unknown) problems
6Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Process quality and product quality
Quality in process -> quality in product
Project: instantiated process
Quality according to ISO 9126
Process quality contributes to improving product quality, which in turn contributes to improving quality in use
Process
Project
7Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Wikipedia says
Quality Assurance (QA) is the activity of providing evidence needed to establish confidence among all concerned, that quality-related activities are being performed effectively.
QA assures the existence and effectiveness of procedures that attempt to make sure - in advance -that the expected levels of quality will be reached.
8Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Definition Quality Assurance IEEE Standard Glossary of Software Engineering Terminology
1. A planned and systematic pattern of all actions necessary to provide adequate confidence that an item or product conforms to established technical requirements.
2. A set of activities designed to evaluate the process by which products are developed or manufactured.
9Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Quality Related Concepts
Runeson and Isacsson, 1998, Software Quality Assurance – Concepts and Misconceptions,in proceedings of Euromicro Conference, pp. 853 - 859.
Software Quality Engineering
Quality AssuranceQuality Development
Software Quality
Assurance
Quality Management
Verification and Validation
Requirements Engineering
System and Software Design
Implementation
10Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Defining the ConceptsSoftware Quality Engineering
Quality AssuranceQuality Development
Software Quality
Assurance
Quality Management
Verification and Validation
Requirements Engineering
System and Software Design
Implementation
Software Quality Engineering (SQE) All activities involved in engineering a software product with respect to its quality
Activities to achieve quality (Quality Development) as well as to assure the required software quality (Quality Assurance)
Software Quality Assurance (SQA)Activities for independent assurance of adherence to defined processed (as stated in CMM)
Quality Management (QM) Project’s internal activities to monitor and manage the software quality
E.g., quality planning, process definitions, process monitoring, …Verification and Validation (V&V)
Concrete activities to verify that the system and its components conforms to their specifications
and to validate that the customer needs are fulfilled
E.g., different types of tests and inspectionsRuneson and Isacsson, 1998, Software Quality Assurance –Concepts and Misconceptions, in proceedings of Euromicro Conference, pp. 853 - 859.
11Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Defect Prevention and Detection (quality development and quality assurance)
Both are needed and must be taken into account
12Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Defect Prevention (quality development, quality building)
Quality practices include all activities that contribute to the quality of the final product
Defect prevention is essentialPreventing defects to occur in the first placeDevelopment practices, conventions, techniques, and tools
Many practices are used to prevent bad quality deliverables
Coding conventions and standardsDesign patternsReviewsCommunication practicesEtc.
Reviews and inspectionsDetect defects in current phasePrevent defects in the following phases
13Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Defect Detection (quality assurance, verification & validation)
Earlier detected defects are cheaper to fix
Prevented defects can be even cheaper
Defect detection is always needed
Nobody is perfect, all defects cannot be prevented
Detection of some bearable amount of defects can be cheaper than prevention
Detection for confidence
Testing and measuring gives information of the achieved quality
We try to find defects in order to convince that there are none
14Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
The Purpose of Quality Assurance
Independent assurance of adherence to defined processes
Creating sufficient and applicable processes and practices for achieving good enough quality level
Monitor the process and ensure that
the selected processes and practices are followeddeviations are reported and acted uponbad practices are improved or replaced by better practices
Support the delivery of good enough quality deliverables
Providing visibility into the achieved level of qualityProviding valuable feedback for people
For improving their working practices
Measuring and tracking quality metrics and process issues
15Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Example: Unit Testing as a quality practice
Unit testing is a quality practice
A developer writes unit tests for all production code
Each source code class must have unit tests and the tests must be automated
All unit tests are reviewed by another developer
All unit tests are run, and must pass, nightly
16Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Example: Unit Testing as part of QA
The role of quality assurance team is toUnderstand the benefits of the unit testing practice
Train and motivate development teams to use the unit testing practice
Assure that each class have unit testsall unit tests have been peer reviewednightly builds are run and pass
Measure and track trends in relevant metricsAmount of unit testsAmount of effort used to reviewing the unit testsResults of the peer reviews (number, type, severity of found issues)Results of the nightly buildsAmount and type of defects that the unit tests catch Amount and type of defects found later, that should have been caught by the unit tests
Help developers to analyse the collected data and improve their unit testing
19Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
CMM Quality Assurance Objectives
Supports the delivery of high-quality products and services
providing visibility into, and feedback on, processes and work productsfor the project staff and managers at all levels
Ensure that the planned processes are implemented
Practices in verification process area ensure that the specifiedrequirements are satisfied
20Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
CMM Quality Assurance is
Objectively evaluating
performed processes
work products
services
Against the applicable
process descriptions
standards
procedures
Identifying and documenting non-compliance issues
Providing feedback on the results to
project staff
managers
Ensuring that non-compliance issues are addressed
21Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
CMM Quality Assurance Organisation
Independence
Independent SQA group
SQA people separate from those directly developing the work product
Reporting
Non-compliance issues primarily solved on a team level
Independent reporting channel to escalate non-compliance issues as necessary
To upper management
22Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
CMM Quality Assurance Goals and Practices
Objectively evaluate processes and work products
Clearly stated evaluation criteriaWhat, when, how, and who must be involved
Evaluation and non-compliance reports
Documented corrective actions
Provide objective insight
Communicating the non-compliance issues
If local (team level) resolution of the issues is not possible ensure that appropriate level of management can resolve the issue
Tracking the non-compliance issues to resolution
Tracking and analysing trends of quality issuesCommunicating the quality trends
24Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
A Different Approach
In agile development methodologies the independent SQA is not so clearly visible
Not a separate organization or role
In agile world the documented process and practices are not emphasized
Individuals are respected and the team can decide their own practices
Quality management on team or project level is emphasized
The Methodology-Growing Technique is one example of agile approach to quality assurance
25Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Methodology-Growing Technique A. Cockburn, Agile Software Development, 2002
“On-the-fly methodology construction and tuning”
Choosing and documenting the practices
Evaluating the practices
Identifying non-compliance issues and bad practice issues
Finding resolutions and tuning the process
Focusing into the team, what was done and how well it worked out, and how to become better
Instead of documents, independent evaluation, and reporting issues
26Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Methodology Growing Technique
Discover the strengths and weaknesses of your organization through short project interviews
1. Create a base methodology at the start of the project
Adjust it together with the development team
2. Run a small interview in the middle of an increment
“Are we going to make it, working the way we are working?”
3. Hold a team reflection workshop after each increment
4. (Post-project review)
27Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Reflection Workshop
A team meeting lasting typically around half a day
Evaluating the effects of each element of the processAfter delivering a running tested incrementOnly then you can see what was underdone and what was overdone
Main questions“What did we learn?”“What can we do better?”
Results can be Tightened standardsNew practices and conventionsChanged practicesDropping unnecessary things outChanges in physical working environmentChanges in team organisation…
28Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Reflection Workshop - Deliverable
Worked well describes the current practices of the team
Problems Lists the identified issues
Try these presents the agreed resolutions for the issues and other improvement actions
Evaluation criteria is not documented
Worked well
Problems
Try these
Reflection Workshop Poster
29Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Choices for (S)QA
OrganisationIndependent QA teamIntegrated into the project team
EnsuringCompliance to documented processesApplicability and improvement of the current processes and practices
Evaluation CriteriaAgainst predefined criteria Identifying issues and problems
FocusDocuments and processes End products and quality
CommunicationReporting to managementSupporting the team
30Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Summary: The Purpose of Software Quality Assurance
Independent assurance of adherence to defined processes
Creating sufficient and applicable processes and practices for achieving good enough quality level
Monitor the process and ensure that
the selected processes and practices are followeddeviations are reported and acted uponbad practices are improved or replaced by better practices
Support the delivery of good enough quality deliverables
Providing visibility into the achieved level of qualityProviding valuable feedback for people
For improving their working practices
Measuring and tracking quality metrics and process issues
32Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Test Oracles – How do we know it’s broken?
An oracle is the principle or mechanism by which you recognize a problem
Test oracle provides the expected result for a test
Specification document
Formula (mathematical)
Computer program
A person
In many cases it is very hard to find an oracle
Even the customer and end user might not be able to tell (beforehand) which is the correct behaviour
Oracle problem is one of the fundamental issues in test automation:How do we teach an automated test to recognize a defect or failure when it happens? – This is not a trivial problem.
33Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
What is a software defect (bug)?
A software defect is present when the program does not do what its end user reasonably expects it to do. (Myers, 1979)
The extent to which a program has bugs is measured by the extent to which it fails to be useful. (Beizer, 1984)
A software defect is a mismatch between the program and its specification.
A defect can be detected if the specification exists and is correct.
34Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Rules for identifying a defect
The software does not do something that the specification says it should do
The software does something that the specification says it should not do
The software does something that the specification does not mention
The software does not do something that the specification does not mention but it should
The software is difficult to understand, hard to use, slow, or will be viewed by the end user as just plain not right
Suorce: Software Testing, Ron Patton
35Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Why defects have to be reported?
To make sure found defects are not lost again (forgotten)
To get found defects fixed
To enable defect managementPrioritizationScheduling for fixingTrade-off decisionsTracking trends
Evaluating qualityEstimating effort and schedules
To support quality assuranceMeasuring defect counts, severity distributions, trends, …Estimating risks
36Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Bugs have a life-cycle
New Assigned
Reopened
Resolved Verified
Closed
Unconfirmed
Open States End States
37Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Reporting the found defects
Report the defects immediately
Don’t leave it until the end of the day
Use some defect management system to report and manage defects (e.g. Bugzilla, Mantis)
Do not manage defect reports in emails, postit’s, or .doc’s
Defect reports are challenging to write
Bearing bad newsExplaining complicated behaviourCommunicating to people with different mindset using as few words as possibleGoal is to make people fix their mess instead of creating some new fancy functionality
In worst case, defect reports are the main communication channel between testing and development
38Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Basic structure of a defect report
1. Defect-report identifier
2. Title: A brief, but descriptive summary
3. Date, time, and who found
4. Test item and environment including version and build numbers
5. Description: A detailed description of the defect
1. Expected results2. Actual results3. Repeatability (how to repeat, if possible)4. Additional information that may help to isolate and
correct the cause of the incident
6. Severity of the defect
39Juha Itkonen, 2008SoberITHELSINKI UNIVERSITY OF TECHNOLOGY
Writing down defects
Understand the defect forstSpend some time to understand what is the actual defect and under which conditions it occursFind out how to reproduce the defectWhat were the specific expected and actual behaviour/values
Be brief Write condensed report with all relevant information but nothingmore
Be clear Avoid ambiguous or misleading terms
Be neutralDon’t be judgemental; reports should be written against the product,not the person, and state only the facts
Be concreteCommunicate the severity as clearly as possibleReal consequences for the users and customer