Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate...

36
Visual GUI Testing Automated System- and Acceptance-Testing Emil Alégroth 2015-02-27

Transcript of Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate...

Page 1: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Visual GUI TestingAutomated System- and Acceptance-Testing

Emil Alégroth2015-02-27

Page 2: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Who am I?

MSc 2010 from ChalmersPhD student since 2011Defended my Licenciate thesis 2013

And passed :P

http://www.cse.chalmers.se/~algeroth/lic.html

Page 3: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Layers of system abstraction

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code

SYSTEMInput Output

Page 4: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Test Automation in practice

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code Mutation analysis/Unit testing/etc.

Unit testing

Model-based testing/Random testing/etc.

Component/Widget based testing

?

System layer viewExample test techniques

Visual GUI Testing (VGT)

Page 5: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Pictorial GUI

GUI Model

System/subsystem architecture

Software components

Source code

Holistic view required!

Unit test: What line(s) of code (almost) where the defect residesbut not what feature that is broken.

VGT: What feature is defective but not what line of codeCombine different approaches: Gives you both the broken feature and potentially line(s) of code.

Page 6: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Regression testing

System

v1

Test suite

v1

System

v2System

v3Time

Test suite

v1Repaired

testsAdditional tests

Test suite

v1Repaired

testsAdditional testsAdditional tests

Purpose of regression testing: Verify that changes to the system has not

broken parts of the system that previously worked.

Implication: Test suite must be maintained to conform to the system’s

requirements.

Page 7: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Manual GUI-based RegressionTesting

Test step

Input Expected output

Result

1 Click on button x

Button x changes color

Yes

2 Click on button y

Button y changes color

.. … … …

N Click on button z

Button z changes color

Yes

Yes

NOOOOOOOOOOOOOOOO!!!!!!

No

Page 8: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Manual GUI-based RegressionTesting

Costly Tedious Error-prone

Page 9: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Regression testing in Practice

Component level (Common): XUnit - automated testing JUnit, CPPUnit, etc.

System level (Not common) Automated integration tools (Often XUnit here as

well)

GUI model (Moderately common) Selenium – Web systems QTP – Desktop

Pictorial GUI (In all companies!) Manual practices!

Page 10: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

GUI model testing vs Pictorial GUI testing

START

STOP

GUI according to GUI model

START

STOP

Pictorial GUI shown to user

GUI model test tools: It looks great!Human: Ehhh? No!

Page 11: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Visual GUI Testing

Test step

Input Expected output

1 Click on button x

Button x changes color

2 Click on button y

Button y changes color

.. … …

N Click on button z

Button z changes color

Transition costly andtedious manual test-scenarios… …using image recognition

and scripts…

…to emulate end-user behaviorfor automated System and

Acceptance testing.

Page 12: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Require-ments

GUI test

System test

Why VGT?

Develop-ment

Commit

Build

Release

The continuous delivery cycle

Unit test

GUI TestManual:CostlyTediousError-prone

Page 13: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Short video demo

Page 14: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Benefits

Speed Test frequency

Test any GUI-based systemFlexibility

Page 15: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Additional Benefits

Identifies new defectsImproved quality

Easy to understand/learn Anyone/Everyone can use it

Page 16: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Maintenance

1 2 3 4 5 6 7 8 9 10 11

20%

7%

ContinuousVGT Maintenance

Page 17: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Drawbacks

Continuous assurance

Requires continuous WORK! BUT still reported as BENEFICIAL!17-62 %

WORST CASE!

Compared to other automatedtest techniques, e.g. unit tests

Page 18: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Recent success storyCustomer

Development company

Every 3-7 months

Manual reproductionStimulation with aVGT Script

Reproduced in < 24 hoursKilled in 7 days

Page 19: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

What was the defect?

Custom menu

Item 1Item 2Item 3Item 4

Memory

Shows that automated tests (VGT) can identify non-frequent, non-

deterministic, defects!

Page 20: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

VGT best practiceManual state-based test scenario

1-to-1 translation

VGT script

Possible but a bad idea!

Page 21: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

VGT best practice

Split into scenarios

Manual state-based test scenario VGT script

Page 22: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Def1():

Def2():

Def3():

Def4():

Def5():

Def6():

Def7():

VGT best practiceMETA Scenarios Reused components Unique code

DefX():

Page 23: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Why is this best practice?

Promotes Reusability: Defs can be reused for new script

development Maintenance: Images and logic in isolated

components

Sikuli This practice is particularly important in Sikuli

since it does not support test suites!

Valid practice in all VGT tools! Also in other automated testing tools!

Practice from software development

SOFTWARE vs TESTWARE

Page 24: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Tools for VGT

Page 25: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

SikuliOpen source VGT tool developed originally at

MIT

Script tool Developed in Jython Supports Python and imports of Java libraries in

its Integrated Development Environment (IDE) Has a Java Api as well Image recognition algorithm developed in C++ Scriptspråk: Python

Page 26: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

IDE

Page 27: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Location(x,y)Notera: y är negativ!

Sikuli specific methods

wait(a,time)wait(time, a)

type(“c”, KEY_CTRL)type(Key.DELETE, KEY_CTRL + KEY_ALT)

type(Key.ENTER)

ENTER, TAB, CTRL, ALT, CMD, etc.

Page 28: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Python Syntax

NOT, AND, OR==, !=, >, <

Page 29: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Imports

Importing script

Imported script. In this case called: scriptA.sikuli

Page 30: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

How do I find information about Sikuli script?

No official API, but sikuli.org has a lot of valuable information.

Instead! Search on Google “Sikuli <what you need>”, e.g. Sikuli click, Sikuli import,

etc.

For advanced script development “Python <what you need>”, e.g. Python import, Python

loops, etc.

For even more advanced scripts “JAVA <what you need>”, t.ex. JAVA I/O, JAVA jar, etc.

I have created a cheatsheet that you can use for basics: http://www.cse.chalmers.se/~algeroth/cheatsheet.html

Page 31: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

VGT demo

Page 32: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Questions?Emil Alégroth: [email protected]

www.sikuli.org

http://www.cse.chalmers.se/~algeroth/cheatsheet.html

Page 33: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Image settings

Page 34: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Example Script

Page 35: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

More advanced settings/scripts

Page 36: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.

Sikuli OCR and Python I/O