ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing...
Transcript of ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing...
![Page 1: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/1.jpg)
ISO/IEC 29119 The New International
Software Testing Standard
Stuart Reid
Testing Solutions Group
117 Houndsditch
London EC3
UK Tel: 0207 469 1500 Fax: 0207 623 8459
www.testing-solutions.com
©Stuart Reid, 2012
![Page 2: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/2.jpg)
Scope
• The purpose of standards
• Overview of ISO 29119
• Applicability
• Recent developments
• Timeline
• Future work
![Page 3: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/3.jpg)
What are standards?
– Guidelines documents as they are not compulsory unless mandated by an individual or an organization
– Agreements because they often reflect a certain level of consensus
“Guideline documentation that reflects agreements on products, practices, or operations by nationally or internationally recognized industrial, professional, trade associations or governmental bodies” - ISO
![Page 4: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/4.jpg)
Pre-standardization?
![Page 5: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/5.jpg)
Why use standards?
• Consumers
– Confidence in compliant products
– Authors provide expertise in standards
• Manufacturers
– Conformance and Marketing
– Safety from liability
– Guidelines on production
• But not ‘Best Practice’…
![Page 6: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/6.jpg)
Quality and Standards
![Page 7: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/7.jpg)
What use are standards?
• Basis for:
– Communication – common terminology
– Professional qualifications
– Certification/compliance schemes
– Benchmark of ‘good industry practice’
– Contracts
– Interoperability and consistency……
Standards describe a current ‘body of knowledge’ that provides the basis for a professional discipline
![Page 8: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/8.jpg)
A Case in Point
![Page 9: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/9.jpg)
Standardization Bodies
![Page 10: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/10.jpg)
ISO/IEC Software Standards
0
20
40
60
80
100
120
140
1987
1988
1989
1990
1991
1992
1993
1994
1995
1996
1997
1998
1999
2000
2001
2002
2003
2004
2005
2006
2007
2008
2009
2010
Published
Maintained
![Page 11: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/11.jpg)
Motivation for ISO 29119
• Conflicts in definitions, processes & procedures
– plethora of standards to be replaced by one
• e.g. IEEE 829, IEEE 1008, BS 7925-1/-2, IEEE 1028
– practitioners do not know which standard to follow
• Lacking in current standards:
– Organizational Testing not covered
• e.g. Test Policy and Organizational Test Strategy
– no Project Test Management
– BS 7925 only covers unit testing
– common functional techniques missing
– poor coverage of non-functional testing
![Page 12: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/12.jpg)
ISO/IEC SC7 – WG26
ISO
TC176
Quality
JTC1
Information Technology
SC7
Software & Systems
Engineering
SWG1 ..................
WG26
Software Testing
Representatives of National
Standards Bodies
SC22
Programming
Languages
SC27
IT Security
SC32
Data Management &
Interchange
IEC
TC56
Dependability
![Page 13: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/13.jpg)
BS 7925-1
BS 7925-2 IEEE 829
Concepts & Vocabulary
Part 1
ISO/IEC 29119 –Structure
Process Assessment
‘Part 5’
Testing Techniques
Part 4
Documentation
Part 3 Part 2
Processes
Actually ISO/IEC 33063
![Page 14: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/14.jpg)
SOFTWARE TESTING CONCEPTS
Scope, Conformance, Normative References
TESTING IN DIFFERENT LIFE CYCLE MODELS
ROLES AND RESPONSIBILITIES IN TESTING
ANNEXES – Metrics, Examples, Bibliography
DEFINITIONS
Part 1: Concepts & Vocabulary
![Page 15: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/15.jpg)
Part 2: Testing Processes
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
![Page 16: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/16.jpg)
TEST MANAGEMENT PROCESSES
ORGANIZATIONAL TEST PROCESS
DYNAMIC TEST PROCESSES
Instantiating Testing Processes
![Page 17: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/17.jpg)
ORGANIZATIONAL TEST PROCESS
TEST MANAGEMENT PROCESSES
TEST
PLANNING
TEST
MONITORING &
CONTROL
TEST
COMPLETION
ORGANIZATIONAL TEST
DOCUMENTATION
FEEDBACK ON ORGANIZATIONAL TEST DOCUMENTATION
TEST PLAN UPDATES
TEST PLAN
TEST COMPLETION REPORT
DYNAMIC TEST
PROCESSES
TEST
MANAGEMENT
PROCESSES
TEST PLAN, TEST COMPLETION REPORT, TEST MEASURES
TEST MEASURES
TEST PLAN, CONTROL DIRECTIVES
TEST PLAN, CONTROL DIRECTIVES
Test Management Processes
![Page 18: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/18.jpg)
Organise Test Plan
Development
Identify & Estimate Risks
Design Test Strategy
Determine Staffing and Scheduling
Document Test Plan
Schedule, Staffing Profile
Test Strategy
Analyzed Risks
Scope
Identify Risk Treatment Approaches
Gain Consensus on Test Plan
Approved Test Plan
Draft Test Plan
Test Plan Publish
Test Plan
Understand Context
Treatment Approaches
Test Planning Process
![Page 19: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/19.jpg)
Test Monitoring & Control Process
Monitor
Report
measures
Control Set-Up
[testing complete]
[testing incomplete]
control directives
test progress
info
test progress info
test plan
test measures
test status report
test control info
...TEST PROCESSES...
Dynamic/Management
![Page 20: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/20.jpg)
DYNAMIC TEST PROCESSES
TEST SPECIFICATION
[NO ISSUES NOTICED]
TEST ENVIRONMENT READINESS REPORT
TEST ENV’T
REQUIREMENTS
INCIDENTREPORT
TEST DESIGN &
IMPLEMENTATION
TEST
ENVIRONMENT
SET-UP
TEST
EXECUTION
TEST
INCIDENT
REPORTING
(LEVEL) TEST MANAGEMENT PROCESS
[ISSUE NOTICED or RETEST RESULT]
TEST RESULTS
CONTROL DIRECTIVES TEST
MEASURES
(LEVEL) TEST PLAN
Dynamic Test Processes
![Page 21: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/21.jpg)
Test Process Descriptions Example – Organizational Test Process
• Each testing process is described using a standard template (following ISO 24774), for example:
• Purpose
– The purpose of the Organizational Test Process is to develop and maintain organizational test specifications, such as the Test Policy and Organizational Test Strategy.
• Outcomes
– As a result of the successful implementation of the Organizational Test Process:
• The organizational test specification is developed based on stakeholder requirements;
• The organizational test specification is agreed by stakeholders;
• The organizational test specification is published and accessible to stakeholders;
• Conformance of stakeholders with the organizational test specification is managed;
• The organizational test specification is reviewed on a periodic basis;
• Minor updates to the organizational test specification are made as necessary.
• Activities and tasks
– The tester shall implement the following activities and tasks in accordance with applicable organization policies and procedures with respect to the Organizational Test Process.
Develop Test Specification
• This activity consists of the following tasks:
– Analyze any relevant source documents and the current testing practices within the organization to identify requirements for the organizational test specification.
– etc.
Gain Consensus on Test Specification
etc.
• Information items
– The output of the Organizational Test Process is the organizational test specification. The Organizational Test Policy and Organizational Test Strategy are typical examples of organizational test specifications.
• Note that ISO 24774 (and this format) has changed since we started, so we have had to rewrite all the process descriptions
![Page 22: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/22.jpg)
Part 3 – Test Documentation
TEST DOCUMENTATION
ANNEXES - EXAMPLES
Scope, Conformance,
Normative References
![Page 23: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/23.jpg)
Part 3: Test Documentation
• Organizational test documentation
– Test policy
– Test strategy
• Project test documentation
– Project test plan
– Test project completion report
• Test Level documentation
– Test plan
– Test specification
– Test results
– Anomaly reports
– Level test status report
– Test environment report
– Test level completion report
• Appendices
– examples of documents at each level of testing
![Page 24: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/24.jpg)
Part 4 – Test Techniques
TEST COVERAGE MEASUREMENT
Scope, Conformance, Normative References
ANNEXE – TESTING OF QUALITY CHARACTERISTICS
ANNEXE – SELECTION OF TECHNIQUES
ANNEXE – TEST TECHNIQUE EFFECTIVENESS
TEST DESIGN TECHNIQUES
BLACK BOX WHITE BOX
![Page 25: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/25.jpg)
Applicability
• Intended to be a generic standard
– support all domains
• Some domains are more standards-oriented, e.g.
– safety-related
– telecoms
• Some domains will become more regulated, e.g.
– financial – banks, stock markets, etc.
• Needs trialling in all domains and project types
– Applicability to sequential, iterative and agile
– Applicability to new and maintenance projects
– Applicability to IS and embedded
– E.g. when wouldn’t you be selecting and using techniques? When wouldn’t you plan?
![Page 26: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/26.jpg)
When can you start using it… and when can you get involved?
• Typical ISO standards take over 7 years
– ISO 12207 was conceived in 1988 and published in 1995 and represents 17,000 person hours (8.5 person years)
• BS 7925-1 & -2 took 8 years to develop
• IEEE estimates 2-4 years to develop a standard, at a cost of between $2,000 and $10,000 per page [1998]
![Page 27: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/27.jpg)
May 10
May 11
May 12
May 13
…
Timelines – all going to plan!
Working Draft (WD) Committee Draft (CD) Draft International Standard (DIS) Final Draft International Standard (FDIS) Final International Standard (FIS)
Parts 2 & 3
Parts 1 & 4
WD CD1
DIS FDIS
FIS
WD CD1
DIS FDIS
FIS
CD2 CD3
CD2 CD3
![Page 28: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/28.jpg)
Do you want to be involved?
• Join the working group
– representing your national standards body
– 6 day meetings, every 6 months
– contribute between meetings
• Join a mirror group
– for your national standards body
• Contribute materials
• Review drafts
![Page 29: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/29.jpg)
Current status
• Drafts are out for review
– Unsurprisingly (for a testing standard) we get plenty of comments (1000s)
• Countries attending meetings include:
– Denmark, India, China, UK, South Africa, Hong Kong, Spain, Germany, USA, Finland, Australia, Korea, Canada, France, Poland, Japan, Russian Federation, Columbia, Sweden, Brazil, Malaysia, New Zealand, Argentina, India, Peru, Colombia, Turkey
• Last meeting – Mumbai, Nov 2011
• Next meeting – Korea, May 2012
![Page 30: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/30.jpg)
Conclusions
• International standard will provide practitioners with guidelines for testing that cover all aspects of the life cycle
– Provides a consistent set of definitions, processes, procedures & techniques for software testing
• Will be adopted by IEEE, BSI, ISO and other national standards bodies
• Currently has representation from 18 nations & is being reviewed by software testing professionals world-wide
• We still need to ensure widespread applicability
![Page 31: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/31.jpg)
Any Questions?
Any Volunteers?
![Page 32: ISO/IEC 29119 - Software · PDF fileISO/IEC 29119 The New International Software Testing Standard Stuart Reid Testing Solutions Group 117 Houndsditch London EC3 UK Tel: 0207 469 1500](https://reader031.fdocuments.us/reader031/viewer/2022012403/5aa096dc7f8b9a8e178e34cc/html5/thumbnails/32.jpg)
For more information
– if interested in trialling the standard on a project, reviewing drafts or writing examples
• http://softwaretestingstandard.org/
– WG26 website
• http://www.jtc1-sc7.org/
– access to official documents released by WG 26