Introduction to automated visual testing
Transcript of Introduction to automated visual testing
![Page 1: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/1.jpg)
Automating visual software testing
Adam CarmiCo-Founder and CTOApplitools
INTRODUCTION TO AUTOMATED VISUAL TESTING
![Page 2: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/2.jpg)
Automating visual software testing
YOU CAN AND SHOULD
AUTOMATE YOUR VISUAL TESTS!
![Page 3: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/3.jpg)
Automating visual software testing
![Page 4: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/4.jpg)
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
![Page 5: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/5.jpg)
Automating visual software testing
WHAT IS VISUAL TESTING?
A quality assurance activity aimed to verify that a Graphical User Interface appears correctly to users
![Page 6: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/6.jpg)
Automating visual software testing
![Page 7: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/7.jpg)
Automating visual software testing
![Page 8: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/8.jpg)
Automating visual software testing
![Page 9: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/9.jpg)
Automating visual software testing
WHY AUTOMATE?The test matrix is too big to cover manually
• Web browsers• Devices• Operating systems• Screen resolutions• Responsive design• L10n• 3rd Party upgrades
![Page 10: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/10.jpg)
Automating visual software testing
WHY AUTOMATE?Mobile apps quality is critical
• Harder to roll back changes• Can’t push daily• Updates take battery and data• Updates are not mandatory
• Higher quality bar
![Page 11: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/11.jpg)
Automating visual software testing
WHY AUTOMATE?Release cycles keep getting shorter
![Page 12: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/12.jpg)
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
![Page 13: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/13.jpg)
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 14: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/14.jpg)
Automating visual software testing
DEMOHELLO WORLD
![Page 15: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/15.jpg)
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 16: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/16.jpg)
Automating visual software testing
![Page 17: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/17.jpg)
Automating visual software testing
![Page 18: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/18.jpg)
Automating visual software testing
![Page 19: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/19.jpg)
Automating visual software testing
![Page 20: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/20.jpg)
Automating visual software testing
![Page 21: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/21.jpg)
Automating visual software testing
AND MORE…
• 1 pixel offsets in element positioning
• Dynamic content
• Moving elements
• Images of different size
• Performance
![Page 22: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/22.jpg)
Automating visual software testing
![Page 23: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/23.jpg)
Automating visual software testing
IMAGE COMPARISON ENGINES
![Page 24: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/24.jpg)
Automating visual software testing
IMAGE MAGICK
• A command line tool for bitmap processing
• Fuzzing is used to eliminate slight color differences
• An error ratio is usually used to determine a match
$ compare –metric AE –fuzz 5% img1.png img2.png diff.png
2246
![Page 25: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/25.jpg)
Automating visual software testing
THE JAVASCRIPT ENGINES• Resemble.js
Pixel by pixel + error ratio + anti-aliasinghttp://huddle.github.io/Resemble.js
• Blink-DiffPixel by pixel + error ratio + anti-aliasing + perceptual color distance computationhttps://github.com/yahoo/blink-diff
• Looks-SamePixel by pixel + perceptual color distance computation + ignore Carethttps://github.com/gemini-testing/looks-same
![Page 26: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/26.jpg)
Automating visual software testing
APPLITOOLS EYES
• Handles anti-aliasing, pixel offsets, color similarity, and image scaling
• Dynamic and moving content
• Compare images of different sizes
• No error ratio configuration required
• Validates full UI pages
• Fast!
• Supports layout / structure matching
![Page 27: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/27.jpg)
Automating visual software testing
DEMOVISUAL LAYOUT MATCHING
![Page 28: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/28.jpg)
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 29: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/29.jpg)
Automating visual software testing
QUICK FEEDBACK TOOLS
MOTIVATION• Get fast feedback on code
changes• Run tests locally in the
background
CONS• Partial coverage
• Chrome, IE?• Limited navigation
• Stale browser versions
SETUP• Render screenshots with a
headless browser• PhatomJS, SlimerJS
• Configuration file driven tests
![Page 30: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/30.jpg)
Automating visual software testing
VISUAL COVERAGE TOOLS
MOTIVATION• Verify that the app / site looks
right in all real execution environments
• Test as many UI states as possible• Execute many tests in parallel
CONS• Requires test infrastructure• Implement and maintain test code
SETUP• Render screenshots using real
browsers on a variety of operating systems and devices in parallel• WebDriver, Grid
• Code driven tests• WebDriver, DSL
![Page 31: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/31.jpg)
Automating visual software testing
SAMPLE TOOLSCODE / SCRIPT● Needle (Python WD)● Gemini (JS DSL)● Selenium Visual Diff (Java WD)● VisualCeption (PHP CodeCeption)● Shoov (JS WebDriverIO)● Vizregress (.NET WD)● Rspec Page Regression (Capibara)● Applitools Eyes (Appium, XCUI, Espresso)● PhantomCSS (JS CasperJS)● Specter (JS DSL)● FBSnapshotTestCase (XCTest)
CONFIGURATION● Viff● CSS Visual Test● Wraith● Dpdxt● Grunt PhotoBox● Snap And Compare● BackstopJS● CSSCritic● Kobold● DiffCop● SUCCSS
![Page 32: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/32.jpg)
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 33: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/33.jpg)
Automating visual software testing
REPORT DIFFERENCESAs files on the file system
![Page 34: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/34.jpg)
Automating visual software testing
REPORT DIFFERENCESAs a gallery
![Page 35: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/35.jpg)
Automating visual software testing
REPORT DIFFERENCESIn your CI build report
![Page 36: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/36.jpg)
Automating visual software testing
HOW DOES IT WORK? Drive the AUT and take screenshots
Compare screenshots with baseline images
Report differences
Update the baseline
![Page 37: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/37.jpg)
Automating visual software testing
UPDATE THE BASELINE
Gemini
![Page 38: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/38.jpg)
Automating visual software testing
UI VERSION CONTROL• Keep CI tests green• Multiple app versions• Revert to previous baselines• A visual change log (documentation)
1 1 1
2
1
2
3 3
2
4
4
![Page 39: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/39.jpg)
Automating visual software testing
![Page 40: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/40.jpg)
Automating visual software testing
DEMOAUTOMATED MAINTENANCE
![Page 41: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/41.jpg)
Automating visual software testing
AGENDA• Why automated visual testing?• Tools & Technology• Where does it fit?• Q & A
![Page 42: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/42.jpg)
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Before committing / merging code:• See how UI changes appear in other execution
environments• Make sure there are no unintended changes
![Page 43: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/43.jpg)
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Visual testing of frontend components by frontend developers
![Page 44: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/44.jpg)
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Regression, Localization, Accessibility, Cross browser/device, responsive design testing and more…
All team members can review and maintain
A collaboration tool
![Page 45: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/45.jpg)
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
Validate your staging deployment using your production deployment as a baseline
![Page 46: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/46.jpg)
Automating visual software testing
WHERE DOES IT FIT?
CODE UNIT INTEGRATION ACCEPTANCE PRODUCTION
No missing resources in productionNo corruption due to 3rd party data, upgrades, hackers, etc.
![Page 47: Introduction to automated visual testing](https://reader035.fdocuments.us/reader035/viewer/2022062218/58f283081a28ab3c2a8b45c7/html5/thumbnails/47.jpg)
Automating visual software testing
Adam Carmi (@carmiadam)Co-Founder and CTO at Applitools
QUESTIONS?