IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK

@jessicard

hi everyone! thanks for having me heretoday i’m going to talk about implementing a visual css testing framework

using

AUTOMATICscreenshot

COMPARISONto catch style

REGRESSIONSusing automatic screenshot comparison to catch style regressions

@jessicardmy name is jessicard on the internet, and jessica in real life

i work at a company called bugsnag here in san franciscobugsnag is an exception monitoring tooli’m a software engineer there, working primarily in ruby and javascriptbut our stack includes lots of languages, including node and go

(we’re hiring!)

we are currently hiringso please get in touch if you’re interested in working on developer tools at a small company!

IMPLEMENTING A VISUAL CSS TESTING FRAMEWORK

so, back to this whole “implementing a visual css testing framework” thingwhat am i even talking about?

well, at bugsnag, we decided we wanted a way to automatically take two screenshotsof our website, at different times in the app

master feature

for example, since we use git, lets say we had a feature branch that we just pushed a commit to,we’d want to take a screenshot of what our homepage looks like on that branchand a screenshot of how our homepage looks on production, or whats currently running on master

with those screenshots, we’d want to spot those differences

and produce some sort of diff image that highlighted the differenceswhy would we want to do this, you might ask? well,

as we all know, writing, reading, and code reviewing CSS can be really intense

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭

😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭😭and even more intense to refactorat bugsnag, we decided to do a huge organizational and code-style refactor of our CSSand we wanted a way to test that our site looked the same, despite changing all of the code

unfortunately, as you can tell, that didn’t always work out for uswe went through many iterations of refactoring,and we realized we needed a tool to help us test our pages automatically

does thisEXIST?

so, we went on the hunt for a way to test our CSSwe wanted to know if there was a tool already built that did what we wantedwithout knowing exactly what it was that we wanted out of this framework

huxley

we first stumbled upon one of facebook’s open source libraries, huxley

“Watches you browse, takes screenshots, tells you when they change”

in huxley’s readme, it says“watches you browse, takes screenshots, tells you when they change”which sounds amazing!

but… i had noticed that it hadn’t been updated in over a yearthat wasn’t promising, but i decided to give it a shot anyway

👎after a good solid day of fiddling around with huxley, it ended up being a bit too buggyit would have random failures, randomly wouldn’t take screenshotsand i started realizing it wasn’t exactly what i was looking for in a css testing tool

Huxley Phantom CSS GhostStory

Cactus Needle

CSSCritic fighting-layout-bugs

sikuli Mogo

Quixotei kept looking, and it turns out there are a lot frameworks that do some sort of css testingthey work in all different ways - some take screenshots, some aren’t even visuali’d definitely recommend checking some of these out to see if they fit what you’re doing in your apps before building your own

WHAT DO I WANT?

but i really started thinking about itwhat was i looking for in a CSS testing framework?what would fit the way bugsnag is built best?i decided i wanted a visual way to test my CSS, with screenshotsrather than writing out tests describing the visuals

this is where i need a disclaimerat bugsnag, our web dashboard is written in railsthis, mixed with the fact i wanted the tests to take screenshots,affected my decisions in what frameworks i wanted to use

we also use git for our source control at bugsnag, and we use it the “github” way

master

what that means is, we have our master branchwhich is always deployable and production ready(or at least it should be)

master feature

and whenever we want to create a feature,we branch off of master until it’s ready, and then we merge it back in

considering the tools we had at our disposal, and after taking a look at some of the screenshotting framework’s source codes,i realized that there wasn’t actually that much code to a lot of the screenshot librariesso I decided why not, i’m just gonna write one myself

PROCESSi came up with a process of how i thought i wanted my tests to work

VISIT1.number one, i wanted a way to somehow automatically visit pages of our site

so the test would, in an actual browser, hit each page of our site on a local server

SCREENSHOT2.

once the test visited the page, i wanted the test to take a screenshot of that page

the important bit for the screenshotis that i’d want it to take a screenshot of the entire page

not just the current viewport of the browser

for example, if a change happened below the fold of our site, but we weren’t taking full page screenshotswe wouldn’t be able to capture that diff

UPLOAD3.next, i’d need somewhere to store these screenshots,and i’d need a way to upload and download these screenshots from that storage area

upload master

screenshot

so, using git, every time i made a push to a branch, i’d upload a screenshot of the current state of each pageincluding our master branch

download master

screenshot

upload feature

screenshot

and, if i had a screenshot already uploaded to our storage area from our master branch,i’d need a way to upload my current features branch to that storage area,and download my already uploaded master screenshot from there

DIFF4.i’d then need a way to make a diff of my screenshots

master feature

i’d want to diff between a screenshot i took on master, downloaded from our storage area,and the newest screenshot i took, on my feature branch

so if i have the previous screenshot i tookwhich would be the current commit on mastervs my branch’s screenshot

and i’d need a way to mark the differences between these two screenshots visually

VIEW5.finally, after i have diffs for my screenshots, i’d need a way to view the diffseven though i’ll have a place to upload them, i need an accessible way for everyone on the project to view the diff screenshots depending on the commit

BUILDINGour framework

so now that we have a plan, we can start building our framework around these things that we need

TESTINGwith rspec

we need a way to write some tests that will run automatically after each pushso we decided to use rspec

rspec

rspec is a testing tool for the ruby programming languagewe already had rspec for tests our rails app, so writing these tests with rspec made sense

we wanted to be able to write specs that looked like thiswhere we would just be able to navigate to a local URL

and save a screenshot of that pagewe also wanted these tests to be separate of our main tests

we also wanted these tests to be separate of our main tests

we pulled out these visual specs into their own rspec tagthat way, these specs wouldn’t run with our main specs when we were running the suite locally, unless we explicitly wanted them tothis also made it so we could break out our specs on our CI

LOCAL speed

we want our visual specs to be separate for a few reasonsnumber one, our local build speedif our local tests were bogged down by waiting for visual specs, that would become a huge issueby having them broken out, we could iterate on our main app specs faster and be able to push more often

CI speed

number two, is speed with our CI testswe wanted our main specs to still be fast on our CI, so that we can merge non-visual pull requests without waiting for our visual specs to finish

CI, or continuous integration, is a way for us to automatically run our test suite when we push things to githuband after our specs run on our CI, we are able to see if our build is passing or not on github

i also learned that github just released a new feature where you can split up your buildsso now, we’ll be able to split out our visual specs from our main specs hereso we can quickly see which build is passing or failing or still runningwithout the builds being combined

buildbox

at bugsnag, we use buildbox for our CI

buildbox allows us to add steps to our teststhat way, we can run our main specs first, apart from our visual specswhen they’re separate, our visual specs don’t slow down our main specsand we can merge non-visual pull requests without waiting for our visual specs to finish

SCREENSHOTwith selenium

next, we needed a way to visit webpages and take screenshots with our rspec testsfor that, we decided to use selenium

selenium

selenium is a tool for automating browsers for testing purposeswe would need to use, specifically, their web driver APIthis would allow us to drive a browser natively on a local or remote machinemore specifically, this provides an API between us, and a browser

browserstack

to use selenium, we’d need to use a service like sauce labs or browserstackwe need to use a service like this because we need access to an actual browsersince we are using a CI, it doesn’t have browsers just built in on the serverwe’d either have to set up our own virtual machines for these browsers or use one of these services. we ended up trying browserstack

before our visual tests, we’d need to start our proxy to browserstack and a forked rails serverand then we’d make an instance of a selenium web driverand then of course, after all of our tests, we would terminate the services

we also had to allow webmock,web mock a library for stubbing and setting expectations on HTTP requests in Ruby,to run real web requestsin order to use our local servers and upload our screenshots

to get our browserstack proxy running, we would just spawn a new browserstack processand terminate the process using its assigned pid

and to get our rails server running, we would spin up a new rails process at port 3000unless one was currently runningand terminate it the same way as our browserstack process

to set up our selenium webdriver, we just have to pass it the capabilities we wantedand a url to hit, which was pointed at browserstack

setting up our selenium driver was easy,but when we were setting it up,we learned some interesting things about taking screenshots with different browsers

with our web driver, we wanted to hit web pages in a real browser

and be able to take screenshots of the full page, not just the current viewport

unfortunately, this feature only worked in firefox, which was not ideal

since internet explorer and chrome didn’t work, we couldn’t really transform this framework we’re making to be used for browser compatibility or anything like thatalthough this wasn’t ideal, for our purposes right now as a refactoring tool, firefox worked fine

DYNAMIC DATA

after writing tests for static pages such as our homepage, we quickly realized that we'd have an issue with the dynamic data on our dashboard

with dynamic data, you can get false positive diffs because data can change between the viewing timesto combat this, we set up fixture data for our rspec tests, and manually adjusted any other data not covered by fixtures using Selenium's Javascript supportso that we don’t get a false positive diff

DIFFINGwith imagemagick

ok, we now have our tests taking screenshotswe need to figure out a way to make a diff between two of our screenshotsimagemagick worked perfectly for this

imagemagick

despite having literally one of the worst sites i’ve ever seen,imagemagick did exactly what we neededimagemagick is a tool to convert, edit, and compose images

imagemagick has a command-line compare tool that, with various options enabled,allows us to shell out and produce diff screenshots based off of two other screenshots

for example, when we’d make a simple change like changing the header

imagemagick would spot those differences

and produce a screenshot like this

imagemagick has a lot of options you can pass to its compare tooland we take advantage of a few of these options in order to make it work

let’s go back to the options we use for a second and go over how they actually work with what we’re doing

imagemagick’s compare tool, from their website, “mathematically and visually annotates the differences between an image and its reconstruction”or, in my terms, takes two images and provides a visual diff

compare lets you provide a metric that outputs to standard error a measure of the differences between images according to the type given metrichere we’re using PAE, where PAE stands for peak absolute we can use the peak absolute to find the size of the “fuzz” factor needed to make all pixels “similar”

1

2so if we had screenshot1 and screenshot2, that were pretty different

it would end up producing a diff like this

and our peak absolute measurement would be outputted as needing a huge fuzz factor to make all pixels similar

the fuzz factor can be important in case we want to ignore pixels which only changed by a small amountwe might want to ignore small changes in case of false positivesfor example, i’ve had false positives before because gradients rendered SLIGHTLY differently between two images

we don’t use this output right now, but it would be important if you wanted to make an assertion in your tests meaningfullike, actually have a failure if a diff was produced by a certain amountwe didn’t end up doing that because it doesn’t necessarily mean something is wrong if a diff is created

a few times we were running our specs, we noticed that diffs weren’t even being produced

we took a look at the screenshots, and realized that they were different heights or sizes for some reason,like if we made a change that accidentally removed the footer.imagemagick wouldn’t let us do a default compare on these images, so we had to use a subimage search

subimage searching is required to have compare search for the best match location of a small image within a larger imagethis option will produce two images (or two frames)the first is the "difference" image and the second will be the "match score" image

the "match-score" image is smaller image containing a pixel for every possible position of the top-left corner of the given sub-imagethe search will try to compare the sub-image at every possible location in the larger imagethis can make sub-image searching very slowas you could guess, the smaller the sub-image is, the faster this search is

that being said, this option doesn’t take effect unless you have two images that are different sizesthis doesn’t happen often to us, so the amount it slows down our visual specs on CI is not meaningfulespecially since those specs aren’t tied to our main specs

another fun thing we ran into was sometimes our screenshots were COMPLETELY different

VS

like, completely different

and imagemagick was NOT into itin fact, it just wouldn’t even give us a diff, because the images were so differentwe found an option called the “dissimilarity-threshold”this threshold determined how different two images could be in order to diff themit defaulted to .2, so we upped that to 1, and it seemed to do the trick

the only caveat to this, as you may have guessed, is doing diffs on completely different images can slow down your tests by a lotlike our previous issue with sub-image searching, this doesn’t seem to happen muchand since the tests are separate from our main specs, we aren’t too worried about it

the last 3 arguments aren’t exciting, these are just where our current screenshot is

where our master screenshot is

and where we want the diff to save to

STORINGwith aws

now that we had our screenshots and diffs,we needed somewhere to throw the screenshots onlineand be able to grab them back out with our rails appso we decided to use aws

aws, or amazon web services, offers cloud storage and has a ruby apiso we’ll store and retrieve our screenshots from one of their buckets

/#{commit-sha}/#{area-of-site}/#{page-name}/

#{image-type}.png

we ended up using a naming pattern of commit-sha, area-of-site, page-name,and image-type

/a1a1a1a/marketing/

index/diff.png

so, for an example, we could have a commit sha of a1a1a1awhere we’re in the marketing part of our siteon the index pageuploading the diff for that page

diff.png current.png master.png

the image-types could be current screenshot we tookor the master screenshot we downloaded from S3or the diff we made from the two screenshots

VIEWINGwith the internet

viewing the screenshots from an amazon bucket was far less than idealso we decided to set up our own custom viewing page in our admin dashboard

we created a page that listed out our current branches with their last 3 commits

and when you would click through, it would show you, by area, all of your screenshots and diffswhich was our end goal!

THE FUTURE

however, our tool is not perfect right nowbut it’s certainly better than what we hadwhich was nothing, but

MEANINGFUL TEST ASSERTIONS

right now, all of our tests pass whether or not there's a diffthey'll only fail if there's an issue executing the testit could be interesting to make our assertions mean something - like fail if there’s a diffwe’d need to think more about that, because a diff doesn’t necessarily mean failure to us

0% DIFFSin the future, it’d be nice to account for 0% diffs

VS

VS

like, maybe we shouldn’t upload a diff image at all if there’s no diff between the imagesthis could save us space on our aws bucketas well as speed up our tests because we’re trying to upload fewer screenshotsand it would make our admin dashboard less noisy

PULL REQUESTS

we also think it’d be nice to automatically link these to a github pull request

so, when a diff is created, maybe automatically attach it onto its relevant pull requestthis one is a little tricky, because maybe we don’t want to create that much noise every time we pushbut it’s an idea

CURRENT commit on

BRANCH

CURRENT commit on

MASTER

DIFF BETWEEN

another thing is that we currently only diff between our current commit on a branchvs our current most recent commit on master

master feature

so, when we push a new commit to our branch, it’ll diff versus the last thing that was pushed to masterthis way, we know what’s changed between the feature that i’m working on and what’s currently running in production

PREVIOUS commit on

MASTER

CURRENT commit on

MASTER

DIFF BETWEEN

it would be nice if we could diff on masterwith the current most recent commitvs its previous commit

master feature

so that way, in case you push a visual change directly to master, you can still see a diff of it

CURRENT commit on

BRANCH

PREVIOUS commit on

BRANCH

DIFF BETWEEN

it would also be nice to see a diff between the previous commit on the current branch as well

master feature

like, if you pushed a commit to your branch that changes some stuff visually,you might want to see a diff between those two, regardless of whats happening on master

i also would really love to get this hooked up to more browsersthat would enable us to make this into a backwards compatibility toolas well as an automatic browser comparisonto make sure things aren’t bonked in ie

thanks!@jessicard

anyway, that’s all i have!feel free to find me after if you want to talk more about css testing frameworks or just want to say hi!thanks for having me!