Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate...
-
Upload
maximilian-jordan -
Category
Documents
-
view
213 -
download
0
Transcript of Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate...
![Page 1: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/1.jpg)
Visual GUI TestingAutomated System- and Acceptance-Testing
Emil Alégroth2015-02-27
![Page 2: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/2.jpg)
Who am I?
MSc 2010 from ChalmersPhD student since 2011Defended my Licenciate thesis 2013
And passed :P
http://www.cse.chalmers.se/~algeroth/lic.html
![Page 3: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/3.jpg)
Layers of system abstraction
Pictorial GUI
GUI Model
System/subsystem architecture
Software components
Source code
SYSTEMInput Output
![Page 4: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/4.jpg)
Test Automation in practice
Pictorial GUI
GUI Model
System/subsystem architecture
Software components
Source code Mutation analysis/Unit testing/etc.
Unit testing
Model-based testing/Random testing/etc.
Component/Widget based testing
?
System layer viewExample test techniques
Visual GUI Testing (VGT)
![Page 5: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/5.jpg)
Pictorial GUI
GUI Model
System/subsystem architecture
Software components
Source code
Holistic view required!
Unit test: What line(s) of code (almost) where the defect residesbut not what feature that is broken.
VGT: What feature is defective but not what line of codeCombine different approaches: Gives you both the broken feature and potentially line(s) of code.
![Page 6: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/6.jpg)
Regression testing
System
v1
Test suite
v1
System
v2System
v3Time
Test suite
v1Repaired
testsAdditional tests
Test suite
v1Repaired
testsAdditional testsAdditional tests
Purpose of regression testing: Verify that changes to the system has not
broken parts of the system that previously worked.
Implication: Test suite must be maintained to conform to the system’s
requirements.
![Page 7: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/7.jpg)
Manual GUI-based RegressionTesting
Test step
Input Expected output
Result
1 Click on button x
Button x changes color
Yes
2 Click on button y
Button y changes color
.. … … …
N Click on button z
Button z changes color
Yes
Yes
NOOOOOOOOOOOOOOOO!!!!!!
No
![Page 8: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/8.jpg)
Manual GUI-based RegressionTesting
Costly Tedious Error-prone
![Page 9: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/9.jpg)
Regression testing in Practice
Component level (Common): XUnit - automated testing JUnit, CPPUnit, etc.
System level (Not common) Automated integration tools (Often XUnit here as
well)
GUI model (Moderately common) Selenium – Web systems QTP – Desktop
Pictorial GUI (In all companies!) Manual practices!
![Page 10: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/10.jpg)
GUI model testing vs Pictorial GUI testing
START
STOP
GUI according to GUI model
START
STOP
Pictorial GUI shown to user
GUI model test tools: It looks great!Human: Ehhh? No!
![Page 11: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/11.jpg)
Visual GUI Testing
Test step
Input Expected output
1 Click on button x
Button x changes color
2 Click on button y
Button y changes color
.. … …
N Click on button z
Button z changes color
Transition costly andtedious manual test-scenarios… …using image recognition
and scripts…
…to emulate end-user behaviorfor automated System and
Acceptance testing.
![Page 12: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/12.jpg)
Require-ments
GUI test
System test
Why VGT?
Develop-ment
Commit
Build
Release
The continuous delivery cycle
Unit test
GUI TestManual:CostlyTediousError-prone
![Page 13: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/13.jpg)
Short video demo
![Page 14: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/14.jpg)
Benefits
Speed Test frequency
Test any GUI-based systemFlexibility
![Page 15: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/15.jpg)
Additional Benefits
Identifies new defectsImproved quality
Easy to understand/learn Anyone/Everyone can use it
![Page 16: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/16.jpg)
Maintenance
1 2 3 4 5 6 7 8 9 10 11
20%
7%
ContinuousVGT Maintenance
![Page 17: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/17.jpg)
Drawbacks
Continuous assurance
Requires continuous WORK! BUT still reported as BENEFICIAL!17-62 %
WORST CASE!
Compared to other automatedtest techniques, e.g. unit tests
![Page 18: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/18.jpg)
Recent success storyCustomer
Development company
Every 3-7 months
Manual reproductionStimulation with aVGT Script
Reproduced in < 24 hoursKilled in 7 days
![Page 19: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/19.jpg)
What was the defect?
Custom menu
Item 1Item 2Item 3Item 4
Memory
Shows that automated tests (VGT) can identify non-frequent, non-
deterministic, defects!
![Page 20: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/20.jpg)
VGT best practiceManual state-based test scenario
1-to-1 translation
VGT script
Possible but a bad idea!
![Page 21: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/21.jpg)
VGT best practice
Split into scenarios
Manual state-based test scenario VGT script
![Page 22: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/22.jpg)
Def1():
Def2():
Def3():
Def4():
Def5():
Def6():
Def7():
VGT best practiceMETA Scenarios Reused components Unique code
DefX():
![Page 23: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/23.jpg)
Why is this best practice?
Promotes Reusability: Defs can be reused for new script
development Maintenance: Images and logic in isolated
components
Sikuli This practice is particularly important in Sikuli
since it does not support test suites!
Valid practice in all VGT tools! Also in other automated testing tools!
Practice from software development
SOFTWARE vs TESTWARE
![Page 24: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/24.jpg)
Tools for VGT
![Page 25: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/25.jpg)
SikuliOpen source VGT tool developed originally at
MIT
Script tool Developed in Jython Supports Python and imports of Java libraries in
its Integrated Development Environment (IDE) Has a Java Api as well Image recognition algorithm developed in C++ Scriptspråk: Python
![Page 26: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/26.jpg)
IDE
![Page 27: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/27.jpg)
Location(x,y)Notera: y är negativ!
Sikuli specific methods
wait(a,time)wait(time, a)
type(“c”, KEY_CTRL)type(Key.DELETE, KEY_CTRL + KEY_ALT)
type(Key.ENTER)
ENTER, TAB, CTRL, ALT, CMD, etc.
![Page 28: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/28.jpg)
Python Syntax
NOT, AND, OR==, !=, >, <
![Page 29: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/29.jpg)
Imports
Importing script
Imported script. In this case called: scriptA.sikuli
![Page 30: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/30.jpg)
How do I find information about Sikuli script?
No official API, but sikuli.org has a lot of valuable information.
Instead! Search on Google “Sikuli <what you need>”, e.g. Sikuli click, Sikuli import,
etc.
For advanced script development “Python <what you need>”, e.g. Python import, Python
loops, etc.
For even more advanced scripts “JAVA <what you need>”, t.ex. JAVA I/O, JAVA jar, etc.
I have created a cheatsheet that you can use for basics: http://www.cse.chalmers.se/~algeroth/cheatsheet.html
![Page 31: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/31.jpg)
VGT demo
![Page 32: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/32.jpg)
Questions?Emil Alégroth: [email protected]
www.sikuli.org
http://www.cse.chalmers.se/~algeroth/cheatsheet.html
![Page 33: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/33.jpg)
Image settings
![Page 34: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/34.jpg)
Example Script
![Page 35: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/35.jpg)
More advanced settings/scripts
![Page 36: Visual GUI Testing. Who am I? MSc 2010 from Chalmers PhD student since 2011 Defended my Licenciate thesis 2013 And passed :P algeroth/lic.html.](https://reader038.fdocuments.us/reader038/viewer/2022110206/56649d0a5503460f949dd26f/html5/thumbnails/36.jpg)
Sikuli OCR and Python I/O