sikuli-harvardmedicalschool
41
PRACTICAL SIKULI USING SCREENSHOTS FOR GUI AUTOMATION AND TESTING Tsung-Hsiang (Sean) Chang MIT CSAIL 1 Collaborators: Tom Yeh, Rob Miller
-
Upload
dinakaran83 -
Category
Documents
-
view
184 -
download
6
Transcript of sikuli-harvardmedicalschool
PRACTICAL SIKULIUSING SCREENSHOTS FOR GUI AUTOMATION AND TESTING
Tsung-Hsiang (Sean) ChangMIT CSAIL
1
Collaborators: Tom Yeh, Rob Miller
WHAT IS SIKULI?2
WHAT IS SIKULI?2
1
2
3
3
Low-level API?Mac’s system design?
4
EMPTY THE TRASH CAN5
OUTLINE
• What Sikuli is
• How Sikuli works
• Sikuli for GUI automation
• Sikuli for GUI testing
• Conclusions
6
How Sikuli Works
7
TEMPLATE MATCHING
click( )
8
TEMPLATE MATCHING
click( )
8
SYSTEM DESIGN
9
1
2
3
DEMO 1:EMPTY TRASH CAN
10
REGION AND MATCH
11
match = region.find( or “text pattern”)
Region
12
• basic attributes: x, y, width, height
• Match is a Region
• Screen is also a Region
• chained methods: Screen(0).find(W).click(X)
REGION
KEY METHODS OF REGION
• find, findAll, exists, wait, waitVanish
• Mouse Actions
click, doubleClick, rightClick, hover, dragDrop
mouseDown, mouseUp
• Keyboard Actions
type(text), paste(text)
keyDown, keyUp
13
EX2: FACEBOOK STALKER
14
EX2: FACEBOOK STALKER
14
RESTRICT SEARCH REGION
15
But the Region constant is bad
16
APP CLASS
17
• firefox = App.open(“Firefox”)
• firefox.focus(), firefox.close()
• firefox.window([n])
• returns the Region of Firefox’s n-th window
FIND IN CERTAIN APP
18
EX3: MUTE SPEAKERS
19
EX3: MUTE SPEAKERS
19
SPATIAL OPERATORS
20
SPATIAL OPERATORS
20
SPATIAL OPERATORS (2)
21
TEXT MATCHING AND RECOGNITION
22
• click(“OK”), find(“Cancel”)
• region.text() - read the text in a region
VISUAL-EVENT-DRIVEN PROGRAMMING
23
• observes visual events in any Region
• Region.onAppear(Pattern, handler)
• Region.onVanish(Pattern, handler)
• Region.onChange(Pattern, handler)
VISUAL-EVENT-DRIVEN PROGRAMMING
24
VISUAL-EVENT-DRIVEN PROGRAMMING
24
VISUAL DICTIONARY
25
GUITESTING
26
VISUAL ASSERTION
• make sure the appearance of visual feedback
27
VISUAL ASSERTION
• make sure the appearance of visual feedback
27
JUNIT INTEGRATION
28
SIKULI IN COMMAND LINE
29
usage:Sikuli-‐IDE [-‐-‐args <arguments>] [-‐h] [-‐r <sikuli-‐file>] [-‐s] [-‐t <sikuli-‐test-‐case>] -‐-‐args <arguments> specify the arguments passed to Jython's sys.argv -‐h,-‐-‐help print this help message -‐r,-‐-‐run <sikuli-‐file> run .sikuli or .skl file -‐s,-‐-‐stderr print runtime errors to stderr instead of popping up a message box -‐t,-‐-‐test <sikuli-‐test-‐case> run .sikuli as a unit test case with junit's text UI runner
CONCLUSIONS
30
PLATFORM INDEPENDENCE
• Works on ANY GUI that can be displayed on Windows/Linux/Mac
• Virtual machines
• Remote desktop
• Mobile simulators: Android, iPhone
• Web: Flash, HTML+Javascript
31
PROGRAM AGAINST UI
32
Sikuli programs are written against the user interface instead of an API
UI: visible, familiar, always exists API: faster, probably more stable
FUTURE WORK
• fast and accurate OCR on screen
• Accessibility API integration
33
Thanks!
http://sikuli.org
New Sikuli X 1.0-rc2 released.
Follow us on twitter @sikuli
Thanks to our funding sources: NSF and Quanta Computer (as a part of TParty project.)
or, just google “sikuli”
34
