Post on 16-Apr-2017
Automating visual software testing
Adam CarmiCo-Founder and CTOApplitools
INTRODUCTION TO AUTOMATED VISUAL TESTING
Automating visual software testing
YOU CAN AND SHOULD
AUTOMATE YOUR VISUAL TESTS!
Automating visual software testing
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
WHY AUTOMATE?The test matrix is too big to cover manually
• Web browsers• Devices• Operating systems• Screen resolutions• Responsive design• L10n• 3rd Party upgrades
Automating visual software testing
WHY AUTOMATE?Mobile apps quality is critical
• Harder to roll back changes• Can’t push daily• Updates take battery and data• Updates are not mandatory
• Higher quality bar
Automating visual software testing
WHY AUTOMATE?Release cycles keep getting shorter
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
DEMOHELLO WORLD
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
Automating visual software testing
AND MORE…
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance
Automating visual software testing
Automating visual software testing
IMAGE COMPARISON ENGINES
Automating visual software testing
IMAGE MAGICK
• A command line tool for bitmap processing
• Fuzzing is used to eliminate slight color differences
• An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
Automating visual software testing
THE JAVASCRIPT ENGINES• Resemble.js
Pixel by pixel + error ratio + anti-aliasinghttp://huddle.github.io/Resemble.js
• Blink-DiffPixel by pixel + error ratio + anti-aliasing + perceptual color distance computationhttps://github.com/yahoo/blink-diff
• Looks-SamePixel by pixel + perceptual color distance computation + ignore Carethttps://github.com/gemini-testing/looks-same
Automating visual software testing
APPLITOOLS EYES
• Handles anti-aliasing, pixel offsets, color similarity, and image scaling
• Dynamic and moving content
• Compare images of different sizes
• No error ratio configuration required
• Validates full UI pages
• Fast!
• Supports layout / structure matching
Automating visual software testing
DEMOVISUAL LAYOUT MATCHING
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
QUICK FEEDBACK TOOLS
MOTIVATION• Get fast feedback on code
changes• Run tests locally in the
background
CONS• Partial coverage
• Chrome, IE?• Limited navigation
• Stale browser versions
SETUP• Render screenshots with a
headless browser• PhatomJS, SlimerJS
• Configuration file driven tests
Automating visual software testing
VISUAL COVERAGE TOOLS
MOTIVATION• Verify that the app / site looks
right in all real execution environments
• Test as many UI states as possible• Execute many tests in parallel
CONS• Requires test infrastructure• Implement and maintain test code
SETUP• Render screenshots using real
browsers on a variety of operating systems and devices in parallel• WebDriver, Grid
• Code driven tests• WebDriver, DSL
Automating visual software testing
SAMPLE TOOLSCODE / SCRIPT● Needle (Python WD)● Gemini (JS DSL)● Selenium Visual Diff (Java WD)● VisualCeption (PHP CodeCeption)● Shoov (JS WebDriverIO)● Vizregress (.NET WD)● Rspec Page Regression (Capibara)● Applitools Eyes (Appium, XCUI, Espresso)● PhantomCSS (JS CasperJS)● Specter (JS DSL)● FBSnapshotTestCase (XCTest)
CONFIGURATION● Viff● CSS Visual Test● Wraith● Dpdxt● Grunt PhotoBox● Snap And Compare● BackstopJS● CSSCritic● Kobold● DiffCop● SUCCSS
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
REPORT DIFFERENCESAs files on the file system
Automating visual software testing
REPORT DIFFERENCESAs a gallery
Automating visual software testing
REPORT DIFFERENCESIn your CI build report
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
Automating visual software testing
UPDATE THE BASELINE
Gemini
Automating visual software testing
UI VERSION CONTROL• Keep CI tests green• Multiple app versions• Revert to previous baselines• A visual change log (documentation)
1 1 1
2
1
2
3 3
2
4
4
Automating visual software testing
Automating visual software testing
DEMOAUTOMATED MAINTENANCE
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Before committing / merging code:• See how UI changes appear in other execution
environments• Make sure there are no unintended changes
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Visual testing of frontend components by frontend developers
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Regression, Localization, Accessibility, Cross browser/device, responsive design testing and more…
All team members can review and maintain
A collaboration tool
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Validate your staging deployment using your production deployment as a baseline
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
No missing resources in productionNo corruption due to 3rd party data, upgrades, hackers, etc.
Automating visual software testing
Adam Carmi (@carmiadam)Co-Founder and CTO at Applitools
QUESTIONS?