UI Test Automation Effectiveness

<Insert Picture Here>

UI Test Automation EffectivenessAlexandre (Shura) IlineJava SE and JavaFX Quality architect.

The following is intended to outline our general product direction. It is intended for information purposes, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions.

The development, release, and timing of any features or functionality described for Oracle's products remains at the sole discretion of Oracle.

This presentation shares experience gotten from testing such products as

JavaFX Authoring Tool

What's effectiveness

• Regular functional testing• Saving human time

• Special groups of tests(Things that could only be done through automated testing)

• Doing it with less human time

UI testing … by Wikipedia

«GUI software testing is the process of testing a product that uses a graphical user interface, to ensure it meets its written specifications.»

UI testing … most often ...

«Checking whether usage of a product UI leads to results expected by the the person who performs testing»

●Start text editor●Push «File/Open»●Verify file chooser directory●Select some file●Verify editor area content●Verify application title●Verify buttons availabilities●....

Sample test scenario

Test automation ... for me

«Building a process which exercises and reports certain product characteristics while run unattended.»

Test automation is like development

• Putting logic in a code• Same lifecycle:

• Requirements• Design• Implementation

• Same set of problems• Bugs • Instabilities

• Scope • defined through test specification/plan

Test automation is not like development

• Big fat dependency – the tested product• vs many libraries and platform

• Many small programs• vs one big program

• Does one thing good – reports status• vs does many thing ... good

• Perfectness is not the goal• other than maintenance cost, ease of use

UI Test automation

• Avoidable• Testing could be done with no automation

• Less obvious• Many misconceptions

• More expensive• Requires specific tools

• Such as the test drivers

Misconceptions

• Automated testing replaces manual testing• it will find all the bugs• it will find more bugs• it does the same amount of work

• Create once use forever• This is easy• This is too hard• No need to create test plan• “Will just buy the XXX tool and it will solve all out

ptoblems”

UI changes

• “Improvements”• Specification/requirement changes• Bug fixes

If UI is not changing ... Why bother testing?

Test base characteristics

• Coverage• How well the test base tests the product?

• Initial creation cost• How expensive it is to initially implement it?

• Sustainability• How expensive it is to support it?

• Reliability• Does it have intermittent failures?

Coverage

• Implementation coverage• Line, block, condition, sequence, ...

• Specification coverage• How well the tests covering the functional specification

• Public API coverage• Whether the public API is tested fully

• UI coverage• Whether the tests cover UI fully

• Combine with bug density• First cover the areas with more bugs

• Combine with code complexity• First cover the more complicated code.

Initial creation

• Record && replay• Inexpensive in creation• Lacks functionality• (More) expensive to support• Disables effective UI test automation

• Coding• Expensive to create• Require experienced engineers• Encourages effective UI testing• (Much) Less expensive to support

Tests code needed to be changed

• Test code needs to be updated every time UI changes

colorComboBox.selectColor(“Grey”)

turns into colorButton.push();...colorChooser.selectColor(Color.GREY);

18

18

Application UI

Product UIProduct UI

19

19

Coordinates

• click(134,32) //selects some record• click(215,122) //hits “Properties”

• sleep(5) //sleeps to let dialog be painted

• click(64,182) //expands color combo• click(235,182) //selects Gray• click(235,212) //hit OK

20

20

Widgets

• Find “Car records” frame• Find table

• Select “1abc234” cell

• Push “Properties” button• Wait for “1abc234” dialog• Select “Gray” color in combo box• Push “OK”

21

21

Prod

uct U

IPr

oduc

t UI

Dom

ain

mod

el

Dom

ain

mod

el

Car record

ColorColorModelModel MakeMake YearYear License plateLicense plateVINVIN

Test

Widgets or coordinates

22

22

UI Primitives

• Find car list frame

CarListFrame list = new CarListFrame()• Open properties dialog for car “1abc234”

CarDialog propDialog = list.carProperties(“1abc234”);

• Set color to gray

propDialog.setColor(Color.GRAY);• Apply changes

propDialog.ok();

23

23

LibraryPr

oduc

t UI

Prod

uct U

ID

omai

nm

odel

Dom

ain

mod

el

Car record

Test libraryTest library


CarListFrame CarDialog

Test

24

24

Domain model

• Set color to gray for a car “1abc234”

new CarRecord(“1abc234”).

setColor(Color.GRAY);

Underneath the cover, CarRecord class does all described earlier

25

25

Domain libraryPr

oduc

tU

IProd

uct

UI

Dom

ain

mod

el

Dom

ain

mod

el Car record

Domain test libraryDomain test library


UI test libraryUI test libraryCarList CarDialog

CarRecord

Test

The formula?

TD + *T

SN

R

TM * N

RN

C*EA =

NC

*

EA – automation effectiveness

To be used for every particular product.

NR and N

C are unique for a product.

TM is a characteristic of a test suite.

Smaller TD and T

S - higher the E

A.

Coefficient depend on the way you write your tests

Coordinates Widgets UI Library Domain library0

1

2

3

4

5

6

7

8

1.1

3

5

7.5

1

0.50.1 0.05

Td/TmTs/Tm

Td and T

s together


5

10

15

20

25

30

1.1

3

5

7.5

24

12

2.41.2

25.1

15

7.48.7

TdTsTd+(Ts*Nc*Nr)

TD and T

S for N

C=3, N

R=8, T

M=1


0.5

1

1.5

2

2.5

3

3.5

0.96

1.6

3.24

2.76

Ea

EA for N

C=3, N

R=8

Other Ts improvements

• Store resources in a property file• Take from the application itself

Testbase reliability

• Select right tool• No sleeps – only waitings• Use event queue• Right lookup criteria

• By ID• By text• By location• Not by index

32

32

Special test types

• Continuous build tests• aka sanity aka acceptance aka smoke

• Pre-integration

• Performance

Development

Continuous build

Commit

BuildExecuted

automaticallyafter commit

Success

Analysis.Rollback!!!

Code changes

NoYes.

PromoteCode is compilableTest further

Continuous build with testing

Commit

Build Success

TestingIs it working?

Passed

Analysis.

Rollback!!!

Code changes

Testchanges

No

No

Test furtherBuild is good

Code line is healthyGo on ...

Yes

Yes. = Compilation

successful

Pre-integration

Commit

Code changes

Testchanges

Testing Passed

No

Yes

How to benefit from UI automation?Use it!

• Sanity• Pre-integration• Attach to bug reports• Make is a quality criteria• Run it for every build• Show it to the boss :)• Don't forget to show it to the dev. team

<Insert Picture Here>

Alexandre (Shura) IlineJava and JavaFX Quality [email protected]

SQE reporting infrastructure improvements

UI Test Automation Effectiveness

Education

Transcript of UI Test Automation Effectiveness