Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

AUTOMATED VISUAL TESTING

WHAT, WHEN & HOW

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?Q & A

WHAT IS VISUAL TESTING?

A quality assurance activity aimed to verify that a

Graphical User Interface appears correctly to users

A VISUAL BUG

AND ANOTHER…

AND YET ANOTHER (IE 8 NOT OK)…

AND YET ANOTHER (IE 11 OK)…

AND THE LAST ONE…

WHY SHOULD IT BE AUTOMATED?

THE TEST MATRIX IS TOO BIG TO APPROXIMATE MANUALLY

Web browsers Devices Operating systems Screen resolutions Responsive designs L10n

WHY SHOULD IT BE AUTOMATED?

WHY SHOULD IT BE AUTOMATED?

NATIVE / HYBRID MOBILE APPS

Harder to roll back changesCan’t push dailyUpdates take battery and data

Higher quality bar

MANUALVISUALTESTINGISERRORPRONE

WHY SHOULD IT BE AUTOMATED?

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

DRIVING AND CAPTURING

OTHER• Wraith (URLs)• PhantomCSS (CJS)• dpdxt (URLs)• Grunt PhotoBox

(URLs)• Snap And Compare

(URLs)• Specter (JS)• Gemini (JS)• Grunt-Vigo (URLs)• BackstopJS (URLs)• Applitools Eyes (All)

• WebDriverCSS (JS)• Fighting Layout Bugs

(Java)• Wraith-Selenium

(Ruby)• Selenium Visual Diff

(Java)• Needle (Python)• VisualCeption (PHP)• Huxley (RP)• Gemini (JS)• Vizregress (.NET)• Visualci (Java)• Viff (JS)• Pix-Diff (JS)• Applitools Eyes (All)

• CSS Critic (URLs)• FBSnapshotTestC

ase (XCTest)• Kobold (folders)• Applitools Eyes

(All)

DRIVINGAND CAPTURINGWITHSELENIUM

(WebDriverCSS)

MANUALLY WITH EYES EXPRESS

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

WHY NOT?

FALSE POSITIVES

ANTI ALIASING 1/2

ANTI ALIASING 2/2

IMAGE SCALING 1/2

IMAGE SCALING 2/2

AND MORE…1 pixel offsets in element positioning

Dynamic contentMoving elementsImages of different sizePerformance

DEMO

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

REPORT DIFFERENCESAs files on the file system (combined with source control)

REPORT DIFFERENCESAs a Gallery (example from Selenium Visual Diff)

THE WORKFLOW Drive the AUT and take screenshots

Compare screenshots with baseline images

Report differences

Update the baseline

UPDATE THE BASELINE

Rename or commit individual image files

GUI (Gemini GUI)

UPDATE THE BASELINE

Overwrite mode

Automatic maintenance (Applitools Eyes)

AGENDA

Why automated visual testing?Tools & TechnologyWhere does it fit?

WHERE DOES IT FIT?

• Components

• Code review

• Developers

• Designers• QA

Visual testing of frontend components by frontend developers

WHERE DOES IT FIT?

• Pages• Page

sections

• Developers

• Designers• QA• Others

Full or partial validation of application screensA collaboration tool

WHERE DOES IT FIT?

• Staging vs. Production

• Ops• QA

Validate your staging deployment using your production deployment as a baseline

WHERE DOES IT FIT?

• Monitoring

• Ops• QA

No missing resources in productionNo corruption due to 3rd party dataNo corruption due to browser / OS upgrades

QUESTIONS?

Adam Carmi (@carmiadam)Co-Founder and VP R&D at Applitools

adam.carmi@applitools.com

THANK YOU