How To Grade Your Selenium Tests

Special Thanks to Dave Haeffner!

http://elementalselenium.com

What We’ll Cover• Good/bad Tests, Testing Rubric

• Good/bad Page Objects, PO Rubric

• Good/Bad Locators, Locator Rubric

• Report Card

• A Concept For Automating This

Who, Me? Okay• Marcus Merrell (@mmerrell)

• Manager, Web Platform Engineering Team at RetailMeNot, inc

• QA Test Automation from 2001-2015

• Selenium/WebDriver since 2007

• The “bass player” in all things

Let’s talk about tests

What Makes a Bad Test?

• Fails For No Good Reason

• Changes in AUT, Need to Update Test(s)

• Slow

• Unreliable

• Hard to Understand and Maintain

What Makes a Good Test?• Written for BDD or xUnit test framework• Test one thing (atomic)• Each test can be run independently (autonomous,

“idempotent”)• Anyone can understand what it is doing (easy to read)• Similar tests are grouped together• Centralized setup/teardown• Uses Page Objects

Some Solid Tests

A Testing Rubric(Each test starts with 100 points, deduct as necessary)

Item Belong in tests? Score

Selenium Commands No -3 per (max of -9)Locators No -2 per (max of -8)Selenium

setup/teardown* No -20

Hard-coded sleeps No -5 per (max of -20)Implicit wait calls No -10Explicit Wait calls No -3

Conditionals No -5 (max of -20)Calls to Page Objects Yes N/A

Assertion(s)* Yes N/A

Let’s talk aboutPage Objects

http://martinfowler.com/bliki/PageObject.html

What Makes a Bad PO?

• Gigantic (i.e., hundreds or thousands of lines of code)

• Over-reaching responsibility

• Contains overly complicated logic

• Return not enough or too much information to the test (leaky abstraction)

• Assertions happening in the PO instead of the test

What Makes a Good PO?

• Contains State (e.g., locators)

• Contains Behavior (e.g., methods to interact with the page)

• Returns some information about the page (e.g., new page object, text from the page, a boolean result for some check, etc. — never a WebElement)

• Verifies page ready state as part of initialization with a found Element

A Page Object Rubric(Each Page Object starts with 100, deduct points as necessary)

Item Belong in POs? Score

> 200 lines of code No -5 per 50 lines over

Assertions No* -5 per (max of -20)

Hard-coded sleeps No -5 per (max of -20)

Implicit wait calls No -10

Explicit waits Yes N/AVerify page ready

state* Yes -8 if not verifying an element

Locators* Yes N/A

http://se.tips/se-waiting-jim-evans

Let’s talk about locators

Locator Strategies• Class• CSS selectors• ID• Link Text• Partial Link Text• Tag Name• XPath

Good locators are:• unique• descriptive• unlikely to change

That rules a few of these out

Locator Strategies• Class• CSS selectors• ID• Link Text• Partial Link Text• Tag Name• XPath

Good locators are:• unique• descriptive• unlikely to change

That rules a few of these out

Locator Strategies• Class

• CSS selectors• ID

• Link Text• Partial Link Text• Tag Name• XPath

Good locators are:• unique• descriptive• unlikely to change

That rules a few of these out

Start with IDs and Classes

Locator Strategies• Class

• CSS selectors

• ID

• Link Text• Partial Link Text• Tag Name• XPath

Good locators are:• unique• descriptive• unlikely to change

That rules a few of these out

Start with IDs and Classes

Use CSS or XPath (with care)

A Locator Rubric(Each locator gets a grade)

Item Letter Grade E.g.,

Dynamic locators D-User account

specific or tied to page render

Tied to page layout D XPath: / / /CSS: > > >, etc.

Text on the page C Link text, page copy

Reasonable traversal B Parent to child w/in

an element nodeUsing semantic

name B+Input labels

(name='username')

Semantic ID AUnique,

descriptive, unlikely to changeNon-unique locator? -2 full letter grades

So…

What’s your testing GPA?

Topic Weight

Tests 20%

Page Objects 30%

Locators 50%

Example ReportSummary

B-Suite Full Suite

Tests 87Page

Objects 62

Locators 90

Example ReportDrill-down

Test Suite Test Name Test PO Locators OverallUser Login Happy Path 87 90 90 89.4User Login Invalid Email 92 78 93 88.3User Login Insecure PW 99 68 91 85.7User Login Mismatched

PW75 56 90 76.8

User Login Missing Req’d 82 34 95 74.1Store Page Offer Sort 91 62 85 79.3Store Page Code-type 83 44 89 74.3Store Page Sale-type 87 62 87 79.5Analytics Outclick 88 N/A N/A 88Analytics Offer Rank 86 N/A N/A 86Analytics Campaign 87 N/A N/A 87

Example ReportTest Level

Test Name Step Locator Score OverallCode-type Login Xpath(‘//root/@home/@login’) D Relative to rootCode-type Search

‘macys.com’Css(‘class: q’) B Semantic CSS

LinkText(‘submit’) C LinkText brittleCode-type Click Macy’s Id(‘macys.com’) A Gold StandardCode-type Filter Codes Id(‘filterCode’) A Gold StandardCode-type Click Top Code Xpath(‘@div[‘code-link’][0]’) C Brittle

Next Steps• Simple to implement, e.g. IntelliJ plug-in

• Event Listener

• Java/JS SDKs in testing

• Reporting needs work (conceptually)

• Service Performance

Thank You!!

Questions?