© 2012GlobalLogic Inc. 1

Taras Lytvyn

http://ami.lnu.edu.ua/kdais/employees.html

https://twitter.com/djlicker

http://testers.lviv.ua/

© 2012GlobalLogic Inc. 2

Taras Lytvyn

Mathematical models and artificial intelligence in

software testing processes

© 2012GlobalLogic Inc. 3

Contents

- What is artificial intelligence?- Problem statement in general- Analysis of recent researches & technologies- Main idea of artificial intelligence approach

- Math model

- Test oracle based on AI

- Result evaluation and classification. Comparison tool algorithm.

© 2012GlobalLogic Inc. 4

What is artificial intelligence ?

Semiotic Biologicalthinking, judgment, language, emotions, creativity, etc.

Neural networks

mental processes

Intellectual behavior

© 2012GlobalLogic Inc. 5

How well an evaluated app. conforms to its specs.

3 stages: - test data generation, - testing, - result evaluation

Regression testing

Problem statement in generalSIMPLE TEST PROCESS

PROBLEM OF HIDDEN ERRORS

© 2012GlobalLogic Inc. 6

Analysis of recent researches & techs

- metric’s analysis- cost of testing- reliability of testing in

general- optimization processes &

data optimization

WHERE AI CAN BE USED ?

© 2012GlobalLogic Inc. 7

Main ideaDecisions during

Regression Testing

MANUAL AUTOMATION

Assert functions (Actual is equal/not equal to expected)

Intellectual automation Decision System during

Regression Testing

© 2012GlobalLogic Inc. 8

Main ideaDecision System is based on artificial intelligent approach and is

build on algorithm with NEURAL NETWORK usage.

Neural Network training process

© 2012GlobalLogic Inc. 9

Main idea

it works ?

Trained neural network that will emulate our

SOFTWARE workability

Testcases

TestedSoftware

Neural Network

outputs

inputs

inputs

© 2012GlobalLogic Inc. 10

Math Modeling usageThe test case execution process should be presented as

Complex Math Model

- Neural Network needs digits (not text)- Input and output data should be normalized in specific way- Math Model will give some limitations

Theorem & LimitationsjiV couldn’t be empty

There is no intermediate step, if a corresponding stage of verification exist

Intermediate stage of verification is possible

)(,:)( mkklVStTCEjiji llji

.1,,...,:)( 1111 lVSVSSTCPjijijijiji lll

.:)( 111 jijijiVSSTCP

Functions

© 2012GlobalLogic Inc. 11

AI Test Oracle Test oracle – is a classifier that shows us

whether test was passed or not

TestedSoftware

Trained Neural

Network

TestCases

ComparisonTool

AlgorithmResult

classifications

Test oracle classify the result of test case execution

© 2012GlobalLogic Inc. 12

Result classification & Comparison ToolThe Comparison Tool is employed as an independent method of

comparing results from neural network and the results of thetested versions of app.

App. result

Neural Network

result Correct Incorrect

Correct 1 True Positive 2 True Negative

Incorrect 4 False Negative 3 False Positive

Comparison of outputsOutput type Same Different

Binary Both correctBoth wrong

ANN correctAPP correct

Continuous Both CorrectBoth wrong

ANN correctAPP correctBoth wrong

© 2012GlobalLogic Inc. 13

ExampleTC id

(№)

TC Title (test case name)

Test steps (test case steps) / Action

Test Verification (test case verification) /

Expected result

1.2 Gross Pay View

1. Input Regular Hours

2. Input Age3. Input Rate of Pay 4. Click Calculate

1. Regular Hours should be displayed

2. Age should be displayed

3. Rate of Pay should be displayed

4. Gross Pay should be displayed

Input Data (test case) should be NORMALIZED !!!

Structure of ANN is based

on Kolmogorov Arnold

approach

Training phase with 50 – 5000 records

© 2012GlobalLogic Inc. 14

- Introduced a mathematical model describing the test case execution for software that is tested- Constructed a new algorithmic model of test oracle based on neural networks- This Model can be used in the processes of regression testing software

- Comparison Tool as result analyzer was provided

© 2012GlobalLogic Inc. 15

< Q & A ? >

© 2012GlobalLogic Inc. 16

Taras Lytvynautomation QC in Global Logic inc.email : taras.lytvyn@globallogic.comweb: http://testers.lviv.ua/automation_blogskype: tasryk