Certified Tester AI Testing Overview of Syllabus

Certified Tester AI Testing

Overview of Syllabus

Version 1.0

International Software Testing Qualifications Board

Certified Tester AI Testing (CT-AI) Overview

v1.0 of 18 2021-10-01

© International Software Testing Qualifications Board

Copyright Notice

Copyright Notice © International Software Testing Qualifications Board (hereinafter called ISTQB®) ISTQB® is a registered trademark of the International Software Testing Qualifications Board. Copyright © 2021, the authors Klaudia Dussa-Zieger (chair), Vipul Kocher, Qin Liu, Stuart Reid, and Adam Leon Smith. All rights reserved. The authors hereby transfer the copyright to the ISTQB®. The authors (as current copyright holders) and ISTQB® (as the future copyright holder) have agreed to the following conditions of use: Extracts, for non-commercial use, from this document may be copied if the source is acknowledged. Any Accredited Training Provider may use this syllabus as the basis for a training course if the authors and the ISTQB® are acknowledged as the source and copyright owners of the syllabus and provided that any advertisement of such a training course may mention the syllabus only after official Accreditation of the training materials has been received from an ISTQB®-recognized Member Board.

Any individual or group of individuals may use this syllabus as the basis for articles and books, if the authors and the ISTQB® are acknowledged as the source and copyright owners of the syllabus.

Any other use of this syllabus is prohibited without first obtaining the approval in writing of the ISTQB®.

Any ISTQB®-recognized Member Board may translate this syllabus provided they reproduce the abovementioned Copyright Notice in the translated version of the syllabus.


v1.0 of 18 2021-10-01


Revision History

Version Date Remarks

1.0 2021/10/01 Release for GA


v1.0 of 18 2021-10-01


Acknowledgements This document was produced by by a team from the International Software Testing Qualifications Board: Klaudia Dussa-Zieger (chair), Werner Henschelchen, Vipul Kocher, Qin Liu, Stuart Reid, Kyle Siemens and Adam Leon Smith. This core team thanks the review team and the Member Boards for their suggestions and input.


v1.0 of 18 2021-10-01


Table of Contents Copyright Notice ...................................................................................................................................... 2 Revision History ....................................................................................................................................... 3 Acknowledgements ................................................................................................................................. 4 Table of Contents .................................................................................................................................... 5 1. Introduction to the AI Testing Syllabus ........................................................................................... 6

1.1 Career Paths for Testers ........................................................................................................ 6 1.2 Intended Audience ................................................................................................................. 6 1.3 Learning Objectives ............................................................................................................... 6 1.4 Entry Requirements ............................................................................................................... 6 1.5 Exam Structure ...................................................................................................................... 6 1.6 Course Duration ..................................................................................................................... 7 1.7 Handling of Standards ............................................................................................................ 7

2. The Certified Tester AI Testing Syllabus ........................................................................................ 8 2.1 Structure and Course Duration .............................................................................................. 8 2.2 Business Outcomes ............................................................................................................... 8 2.3 Content ................................................................................................................................. 10 2.4 Business Outcomes Traceability Matrix with Learning Objectives ...................................... 11

3. References .................................................................................................................................... 18 3.1 ISTQB® Documents ............................................................................................................. 18


v1.0 of 18 2021-10-01


1. Introduction to the AI Testing Syllabus

This overview document is intended for anyone with an interest in the ISTQB® Certified Tester who wants an overview of the Certified Tester AI Testing (CT-AI) Syllabus. In this document the CT-AI module is described in summary form and the business outcomes are stated. These provide a specific statement of what can be expected from a person who achieves the CT-AI Certification and will particularly benefit companies that are considering the development of specific skills at this level.

1.1 Career Paths for Testers

The ISTQB® scheme provides support for the definition of career paths in testing by offering a 3-tiered certification scheme starting with the Foundation Level and continuing with the Advanced Level and Expert Level. These are supported by a collection of Agile modules as well as Specialist modules which enable additional specialist skills to be developed in certain subjects, e.g., AI testing. The Specialist syllabi build on the Foundation Level and establish a platform from which further skills and knowledge may be acquired for different testing topics. Please visit www.istqb.org for the latest overview of ISTQB´s career paths.

1.2 Intended Audience

The Certified Tester AI Testing is suitable for anyone who is involved in testing as well as anyone interested in AI-based systems. This includes people performing activities such as test analysis, test consulting and software development. The syllabus provides testing knowledge for anyone working with Agile or sequential software development lifecycles.

1.3 Learning Objectives

The knowledge levels of the specific learning objectives at K2, K3 and K4 levels are shown at the beginning of each chapter and are classified as follows:

• K1: Remember

• K2: Understand

• K3: Apply

• K4: Analyze The definitions of all terms listed as keywords just below the chapter headings shall be remembered (K1), even if not explicitly mentioned in the learning objectives.

1.4 Entry Requirements

The entry criterion for taking the Certified Tester AI Testing exam is that candidates have acquired the ISTQB® Certified Tester Foundation Level certification [ISTQB_FL_SYL].

1.5 Exam Structure

The Certified Tester AI Testing exam structure is defined in the document “ISTQB_Exam-Structure-Tables_v1.2.xlsx”, which can be found on www.istqb.org.

http://www.istqb.org/


v1.0 of 18 2021-10-01


The Certified Tester Specialist AI Testing module has the following attributes:

• The format of the exam is multiple choice.

• Exam duration is 60 minutes. If the candidate’s native language is not the examination language, the candidate is allowed an additional 25% (exam duration = 75 minutes).

• There are 40 questions.

• To pass the exam, at least 65% of the total sum of points must be answered correctly.

• The total number of points for this exam should be set at 47 points. Therefore, a minimum of 31 points is required to achieve a passing score.

Exams may be taken as part of an accredited training course or taken independently (e.g., at an exam center or in a public exam). Completion of an accredited training course is not a pre-requisite for the exam.

1.6 Course Duration

For accredited training courses, a minimum of 25,1 hours of instruction time is required. Individual training times for each chapter are provided in section 1.1.

1.7 Handling of Standards

There are standards referenced in the Certified Tester Specialist AI Testing syllabus (e.g., ISO, IEC). The purpose of these references is to provide a framework (as in the references to ISO 25010 regarding quality characteristics) or to provide a source of additional information if desired by the reader. Please note that the syllabus uses the standards as a reference. The standards are not intended for examination.


v1.0 of 18 2021-10-01


2. The Certified Tester AI Testing Syllabus

2.1 Structure and Course Duration

The Certified Tester Specialist AI Testing syllabus contains eleven chapters covering the knowledge necessary to be an AI Testing Specialist. The top-level heading for each chapter specifies the minimum time for the chapter; timing is not provided below chapter level. For accredited training courses, the syllabus requires a minimum of 25,1 hours of instruction, distributed across the eleven chapters as follows:

• Chapter 1: 105 minutes Introduction to AI

• Chapter 2: 105 minutes Quality Characteristics for AI-Based Systems

• Chapter 3: 145 minutes Machine Learning (ML) – Overview

• Chapter 4: 230 minutes ML – Data

• Chapter 5: 120 minutes ML Functional Performance Metrics

• Chapter 6: 65 minutes ML – Neural Networks and Testing

• Chapter 7: 115 minutes Testing AI-Based Systems Overview

• Chapter 8: 150 minutes Testing AI-Specific Quality Characteristics

• Chapter 9: 245 minutes Methods and Techniques for the Testing of AI-Based Systems

• Chapter 10: 30 minutes Test Environments for AI-Based Systems

• Chapter 11: 195 minutes Using AI for Testing

2.2 Business Outcomes

This section lists the Business Outcomes expected of a candidate who has achieved the Certified Tester Specialist AI Testing certification. A Certified Tester Specialist AI Testing can:

BO Id BO Text

AI 1 Understand the current state and expected trends of AI.

AI 2 Experience the implementation and testing of a ML model and recognize where testers can best influence its quality.

AI 3 Understand the challenges associated with testing AI-Based systems, such as their self-learning capabilities, bias, ethics, complexity, non-determinism, transparency and explainability.


v1.0 of 18 2021-10-01


BO Id BO Text

AI 4 Contribute to the test strategy for an AI-Based system.

AI 5 Design and execute test cases for AI-based systems.

AI 6 Recognize the special requirements for the test infrastructure to support the testing of AI-based systems.

AI 7 Understand how AI can be used to support software testing.


v1.0 of 18 2021-10-01


2.3 Content

Chapter 1: Introduction to AI

• Definition of AI and AI Effect

• Narrow, General and Super AI

• AI-based and Conventional Systems

• AI Technologies

• AI Development Frameworks

• Hardware for AI-Based Systems

• AI as a Service (AIaaS)

• Pre-Trained Models

• Standards, Regulations and AI

Chapter 2: Quality Characteristics for AI-Based Systems

• Flexibility and Adaptability

• Autonomy

• Evolution

• Bias

• Ethics

• Side Effects and Reward Hacking

• Transparency, Interpretability and Explainability

• Safety and AI Chapter 3: Machine Learning (ML) – Overview

• Forms of ML

• ML Workflow

• Selecting a Form of ML

• Factors Involved in ML Algorithm Selection

• Overfitting and Underfitting Chapter 4: ML – Data

• Data Preparation as Part of the ML Workflow

• Training, Validation and Test Datasets in the ML Workflow

• Dataset Quality Issues

• Data Quality and its Effect on the ML Model

• Data Labelling for Supervised Learning Chapter 5: ML Functional Performance Metrics

• Confusion Matrix

• Additional ML Functional Performance Metrics for Classification, Regression and Clustering

• Limitations of ML Functional Performance Metrics

• Selecting ML Functional Performance Metrics

• Benchmark Suites for ML Performance Chapter 6: ML – Neural Networks and Testing

• Neural Networks

• Coverage Measures for Neural Networks

Chapter 7: Testing AI-Based Systems Overview

• Specification of AI-Based Systems

• Test Levels for AI-Based Systems


v1.0 of 18 2021-10-01


• Test Data for Testing AI-Based Systems

• Testing for Automation Bias in AI-Based Systems

• Documenting an AI Component

• Testing for Concept Drift

• Selecting a Test Approach for an ML System Chapter 8: Testing AI-Specific Quality Characteristics

• Challenges Testing Self-Learning Systems

• Testing Autonomous AI-Based Systems

• Testing for Algorithmic, Sample and Inappropriate Bias

• Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems

• Challenges Testing Complex AI-based Systems

• Testing the Transparency, Interpretability and Explainability of AI-Based Systems

• Test Oracles for AI-Based Systems

• Test Objectives and Acceptance Criteria Chapter 9: Methods and Techniques for the Testing of AI-Based Systems

• Adversarial Attacks and Data Poisoning

• Pairwise Testing

• Back-to-Back Testing

• A/B Testing

• Metamorphic Testing (MT)

• Experience-based testing of AI-based Systems

• Selecting Test Techniques for AI-based Systems Chapter 10: Test Environments for AI-Based Systems

• Test Environments for AI-Based Systems

• Virtual Test Environments for Testing AI-Based Systems Chapter 11: Using AI for Testing

• AI Technologies for Testing

• Using AI to Analyze Reported Defects

• Using AI for Test Case Generation

• Using AI for the Optimization of Regression Test Suites

• Using AI for Defect Prediction

• Using AI for Testing User Interfaces

2.4 Business Outcomes Traceability Matrix with Learning Objectives

The following tables show information about Learning Objectives (LO) and Hands-on Objectives (HO) and their coverage of the Business Objectives. The tables contain the following information:

• Section of syllabus (number and title)

• LO/HO number

• K/HO-Level

• Description of LO/HO

• Mapping of LO/HO to Business Objectives. Mapping to AI Business Objectives


v1.0 of 18 2021-10-01


AI 1 AI 2 AI 3 AI 4 AI 5 AI 6 AI 7

1. Introduction to AI

1.1. Definition of AI and AI Effect

AI-1.1.1 K2 Describe the AI effect and show how it influences the definition of AI.

x

1.2. Narrow, General and Super AI

AI-1.2.1 K2 Distinguish between narrow AI, general AI, and super AI.

x

1.3. AI-Based and Conventional Systems.

AI-1.3.1 K2 Differentiate between AI-based systems and conventional systems.

x

1.4 AI Technologies

AI-1.4.1 K1 Recognize the different technologies used to implement AI.

x

1.5 AI Development Frameworks

AI-1.5.1 K1 Identify popular AI development frameworks. x

1.6 Hardware for AI-Based Systems

AI-1.6.1 K2 Compare the choices available for hardware to implement AI-based systems.

x

1.7 AI as a Service (AIaaS)

AI-1.7.1 K2 Explain the concept of AI as a Service (AIaaS). x

1.8 Pre-Trained Models

AI-1.8.1 K2 Explain the use of pre-trained AI models and the risks associated with them.

x

1.9 Standards, Regulations and AI

AI-1.9.1 K2 Describe how standards apply to AI-based systems.

x

2. Quality Characteristics for AI-Based Systems

2.1 Flexibility and Adaptability

AI-2.1.1 K2 Explain the importance of flexibility and adaptability as characteristics of AI-based systems.

x

2.2 Autonomy

AI-2.2.1 K2 Explain the relationship between autonomy and AI-based systems.

x

2.3 Evolution

AI-2.3.1 K2 Explain the importance of managing evolution for AI-based systems.

x

2.4 Bias

AI-2.4.1 K2 Describe the different causes and types of bias found in AI-based systems.

x

2.5 Ethics

AI-2.5.1 K2 Discuss the ethical principles that should be respected in the development, deployment and use of AI-based systems.

x

2.6 Side Effects and Reward Hacking


v1.0 of 18 2021-10-01


AI-2.6.1 K2 Explain the occurrence of side effects and reward hacking in AI-based systems.

x

2.7 Transparency, Interpretability and Explainability

AI-2.7.1 K2 Explain how transparency, interpretability and explainability apply to AI-based systems.

x

2.8 Safety and AI

AI-2.8.1 K1 Recall the characteristics that make it difficult to use AI-based systems in safety-related applications.

x

3. Machine Learning (ML) - Overview

3.1 Forms of Machine Learning

AI-3.1.1 K2 Describe classification and regression as part of supervised learning.

x

AI-3.1.2 K2 Describe clustering and association as part of unsupervised learning.

x

AI-3.1.3 K2 Describe reinforcement learning. x

3.2 ML Workflow

AI-3.2.1 K2 Summarize the workflow used to create an ML system.

x

3.3 Selecting an ML Approach

AI-3.3.1 K3

Given a project scenario, identify an appropriate ML approach (from classification, regression, clustering, association, or reinforcement learning).

x

3.4 Factors involved in ML Algorithm Selection

AI-3.4.1 K2 Explain the factors involved in the selection of ML algorithms.

x

3.5 Overfitting and Underfitting

AI-3.5.1 K2 Summarize the concepts of underfitting and overfitting.

x

HO-3.5.1 H0 Demonstrate underfitting and overfitting. x

4. ML - Data

4.1 Data Preparation as part of the ML Workflow

AI-4.1.1 K2 Describe the activities and challenges related to data preparation.

x

HO-4.1.1 H2 Perform data preparation in support of the creation of an ML model.

x

4.2 Training, Validation and Test Datasets in the ML Workflow

AI-4.2.1 K2 Contrast the use of training, validation and test datasets in the development of an ML model.

x

HO-4.2.1 H2 Identify training and test datasets and create an ML model.

x

4.3 Dataset Quality Issues

AI-4.3.1 K2 Describe typical dataset quality issues. x

4.4 Data quality and its effect on the ML model


v1.0 of 18 2021-10-01


AI-4.4.1 K2 Recognize how poor data quality can cause problems with the resultant ML model.

x

4.5 Data Labelling for Supervised Learning

AI-4.5.1 K1 Recall the different approaches to the labelling of data in datasets for supervised learning.

x

AI-4.5.2 K1 Recall reasons for the data in datasets being mislabelled.

x

5. ML Functional Performance Metrics

5.1 Confusion Matrix

AI-5.1.1 K3 Calculate the ML functional performance metrics from a given set of confusion matrix data.

x

5.2 Additional ML Functional Performance Metrics for Classification, Regression and Clustering

AI-5.2.1 K2

Contrast and compare the concepts behind the ML functional performance metrics for classification, regression and clustering methods.

x

5.3 Limitations of ML Functional Performance Metrics

AI-5.3.1 K2 Summarize the limitations of using ML functional performance metrics to determine the quality of the ML system.

x

5.4 Selecting ML Functional Performance Metrics

AI-5.4.1 K4 Select appropriate ML functional performance metrics and/or their values for a given ML model and scenario.

x

HO-5.4.1 H2 Evaluate the created ML model using selected ML functional performance metrics.

x

5.5 Benchmark Suites for ML

AI-5.5.1 K2 Explain the use of benchmark suites in the context of ML.

x

6. ML - Neural Networks and Testing

6.1 Neural Networks

AI-6.1.1 K2 Explain the structure and function of a neural network including a DNN.

x

HO-6.1.1 H1 Experience the implementation of a perceptron.

x

6.2 Coverage Measures for Neural Networks

AI-6.2.1 K2 Describe the different coverage measures for neural networks.

x

7. Testing AI-Based Systems Overview

7.1 Specification of AI-Based Systems


v1.0 of 18 2021-10-01


AI-7.1.1 K2 Explain how system specifications for AI-based systems can create challenges in testing.

x x

7.2 Test Levels for AI-Based Systems

AI-7.2.1 K2 Describe how AI-based systems are tested at each test level.

x x

7.3 Test Data for Testing AI-Based Systems

AI-7.3.1 K1 Recall those factors associated with test data that can make testing AI-based systems difficult.

x x

7.4 Testing for Automation Bias in AI-Based Systems

AI-7.4.1 K2 Explain automation bias and how this affects testing.

x x

7.5 Documenting an ML Model

AI-7.5.1 K2

Describe the documentation of an AI component and understand how documentation supports the testing of AI-based systems.

x x

7.6 Testing for Concept Drift

AI-7.6.1 K2 Explain the need for frequently testing the trained model to handle concept drift.

x x

7.7 Selecting a Test Approach for an ML System

AI-7.7.1 K4 For a given scenario determine a test approach to be followed when developing an ML system.

x x

8. Testing AI-Specific Quality Characteristics

8.1 Challenges Testing Self-Learning Systems

AI-8.1.1 K2 Explain the challenges in testing created by the self-learning of AI-based systems.

x

8.2 Testing Autonomous AI-Based Systems

AI-8.2.1 K2 Describe how autonomous AI-based systems are tested

x

8.3 Testing for Algorithmic, Sample and Inappropriate Bias

AI-8.3.1 K2 Explain how to test for bias in an AI-based system.

x

8.4 Challenges Testing Probabilistic and Non-Deterministic AI-Based Systems

AI-8.4.1 K2 Explain the challenges in testing created by the probabilistic and non-deterministic nature of AI-based systems.

x

8.5 Challenges Testing Complex AI-based Systems

AI-8.5.1 K2 Explain the challenges in testing created by the complexity of AI-based systems.

x

8.6 Testing the Transparency, Interpretability and Explainability of AI-based Systems


v1.0 of 18 2021-10-01


AI-8.6.1 K2 Describe how the transparency, interpretability and explainability of AI-based systems can be tested.

x

HO-8.6.1 H2 Use a tool to show how explainability can be used by testers.

x

8.7 Test Oracles for AI-Based Systems

AI-8.7.1 K2 Explain the challenges in creating test oracles resulting from the specific characteristics of AI-based systems.

x

8.8 Test Objectives and Acceptance Criteria

AI-8.8.1 K4 Select appropriate test objectives and acceptance criteria for the AI-specific quality characteristics of a given AI-based system.

x

9. Methods and Techniques for the Testing of AI-Based Systems

9.1 Adversarial Attacks and Data Poisoning

AI-9.1.1 K2 Explain how the testing of ML systems can help prevent adversarial attacks and data poisoning.

x

9.2 Pairwise Testing

AI-9.2.1 K2 Explain how pairwise testing is used for AI-based systems.

x

HO-9.2.1 H2 Apply pairwise testing to derive and execute test cases for an AI-based system.

x

9.3 Back-to-Back Testing

AI-9.3.1 K2 Explain how back-to-back testing is used for AI-based systems.

x

9.4 A/B Testing

AI-9.4.1 K2 Explain how A/B testing is applied to the testing of AI-based systems.

x

9.5 Metamorphic Testing

AI-9.5.1 K3 Apply metamorphic testing for the testing of AI-based systems.

x

HO-9.5.1 H2 Apply metamorphic testing to derive test cases for a given scenario and execute them.

x

9.6 Experience-Based Testing of AI-Based Systems

AI-9.6.1 K2 Explain how experience-based testing can be applied to the testing of AI-based systems.

x

HO-9.6.1 H2 Apply exploratory testing to an AI-based system.

x

9.7 Selecting Test Techniques for AI-Based Systems

AI-9.7.1 K4 For a given scenario, select appropriate test techniques when testing an AI-based system

x

10. Test Environments for AI-Based Systems


v1.0 of 18 2021-10-01


10.1 Test Environments for AI-Based Systems

AI-10.1.1 K2 Describe the main factors that differentiate the test environments for AI-based systems from those required for conventional systems.

x

10.2 Virtual Test Environments for Testing AI-Based Systems

AI-10.2.1 K2 Describe the benefits provided by virtual test environments in the testing of AI-based systems

x

11. Using AI for Testing

11.1 AI Technologies for Testing

AI-11.1.1 K2 Categorize the artificial intelligence technologies used in software testing.

x

HO-11.1.1 H2 Discuss, using examples, those activities in testing where AI is less likely to be used.

x

11.2 Using AI to Analyze Reported Defects

AI-11.2.1 K2 Explain how AI can assist in supporting the analysis of new defects.

x

11.3 Using AI for Test Case Generation

AI-11.3.1 K2 Explain how AI can assist in test case generation.

x

11.4 Using AI for the Optimization of Regression Test Suites

AI-11.4.1 K2 Explain how AI can assist in optimization of regression test suites.

x

11.5 Using AI for Defect Prediction

AI-11.5.1 K2 Explain how AI can assist in defect prediction. x

HO-11.5.1 H2 Implement a simple AI-based defect prediction system.

x

11.6 Using AI for Testing User Interfaces

AI-11.6.1 K2 Explain the use of AI in testing user interfaces. x


v1.0 of 18 2021-10-01


3. References

3.1 ISTQB® Documents

ID Document name

[ISTQB_FL_SYL] Foundation Level Syllabus, Version 2018 V3.1

Certified Tester AI Testing Overview of Syllabus

Documents

Transcript of Certified Tester AI Testing Overview of Syllabus