UI testing iOS apps for humans

Jakub TurekJuly 24, 2017

EL Passion

About me

Jakub Turek

https://jakubturek.com @KubaTurek turekj

1

Agenda

1. Introduction to UI testing.2. Pain points.3. Apple’s built-in XCUITest.4. Snapshot tests with FBSnapshotTestCase.5. Interaction tests with EarlGrey.6. Combining snapshot & interaction testing.

2

Introduction to UI testing

Definition

UI Testing“UI testing gives you the ability to find and interact with the UIof your app in order to validate the properties and state of theUI elements.”Apple Developer

3

Motivation

Unit test UI testGrantee Developer End userInteracts Source code UI controls

- Code works as intended - Feature presenceProves - The API is simple - Correct output

- Proper layout

4

Writing a first UI test

Test case

Tap Animate progress Highlight

−→ −→

5

UI testing problems (1/2)

What should I assert?

• Texts.• Icons.• Colors.• No progress bars.• Everything.

6

UI testing problems (2/2)

When should I assert?

1. Wait & sleep.2. Tests failed.3. More wait & even more sleep.4. Tests succeeded.

7

My initial impressions

Could we hire the Test AutomationEngineer, please?

8

iOS built-ins

A little bit of history

9

Automation quiz (1/2)

QuizHow Automation test cases were programmed?

10

Automation quiz (2/2)

AnswerUsing JavaScript, obviously.

var main = target.frontMostApp().mainWindow();var field = main.textFields()["Field"];

if (field.value() != "Input")UIALogger.logFail("Expected Field to have Input value");

elseUIALogger.logPass("Field has Input value");

11

XCUITest

• Separate test target.• Swift & ObjC APIs forquerying UI elements.

• Black-box testing.• Two apps:

• app under test,• driver.

12

Test case reminder

Tap Animate progress Highlight

−→ −→

13

XCUITest code

override func setUp() {super.setUp()XCUIApplication().launch()

}

func testNegativeAnswerHighlight() {let app = XCUIApplication()

app.staticTexts["Not really"].tap()

let ticked = app.staticTexts["x Not really"]let predicate = NSPredicate(format: "exists == 1")

expectation(for: predicate, evaluatedWith: ticked, handler: nil)waitForExpectations(timeout: 5.0, handler: nil)

}

14

XCUITest issues (1/2)

Lots of „minor” issues:

• Super slow execution (≈ 10s startup penalty).• Fragile time-based expectations.• Type unsafe NSPredicate API.• Lack of communication with application sources:

• Passing launch arguments.

• Needs workaround for animations with .repeat option.

15

XCUITest issues (2/2)

It is impossible to test the presented case.

protocol XCUIElementAttributes {var identifier: String { get }var frame: CGRect { get }var value: Any? { get }var title: String { get }var label: String { get }var elementType: XCUIElementType { get }var isEnabled: Bool { get }var horizontalSizeClass: XCUIUserInterfaceSizeClass { get }var verticalSizeClass: XCUIUserInterfaceSizeClass { get }var placeholderValue: String? { get }var isSelected: Bool { get }

}

16

Snapshot testing

Snapshot testing

Snapshot testingSnapshot testing compares rendered view to a “referenceimage” stored in the source code repository.

17

Snapshot testing workflow

Snapshot testing workflow is “different”:

1. Building a view:• This means no TDD.

2. Recording a reference snapshot.3. Assertion: comparing the view against the reference.

18

FBSnapshotTestCase

facebook/ios-snapshot-test-case

FBSnapshotTestCase implements:

• Capturing reference snapshots.• Image comparison algorithm.• Visual diffing for expected/actual snapshots.

19

FBSnapshotTestCase - example

class ViewControllerSnapshotTests: FBSnapshotTestCase {override func setUp() {super.setUp()recordMode = false // assertion, true for capturing

}

func testViewControllerHasCorrectView() {let sut = ViewController(nibName: nil, bundle: nil)sut.view.frame = UIScreen.main.bounds

FBSnapshotVerifyView(sut.view)}

}

20

Visual diff

Expected Got

Diff

21

Device agnostic snapshots

• The tests would fail if executed with different screensizes/densities.

• Device agnostic snapshots compare snapshots onmatching devices.

override func setUp() {super.setUp()isDeviceAgnostic = true // record/verify device agnosticrecordMode = true

}

22

Using FBSnapshotTestCase with Quick

ashfurrow/Nimble-Snapshots

describe("view") {it("has a valid snapshot") {let sut = ViewController(nibName: nil, bundle: nil)sut.view.frame = UIScreen.main.bounds

expect(sut.view).to(haveValidSnapshot()) // recordSnapshot()}

}

23

Use cases

Snapshot testing works great for enforcing correctness of:

• Localization & internationalization.• UI edge cases.• Layout for multiple devices, size classes.

24

Snapshot testing issues

Biggest pain points:

• TDD workflow is not possible.• Tests can go red even though the tested view looks asexpected.

25

Interaction testing with EarlGrey

EarlGrey

EarlGreyiOS UI Automation testing framework from Google. It isinspired by Espresso for Android.

26

Key characteristics

Different approach than XCUITest:

• White-box testing:• Matchers work on arbitrary properties.• No need to set accessibility labels.

• Auto synchronization:• Assertions on a stable-state views only.• Expect/wait no more.• Synchronizes with networking, animations and operationsby default.

• Objective-C bridged API for Swift.

27

Example test (1/2)

−→log in

28

Example test (2/2)

func testLoginWorks() {EarlGrey.select(grey_kindOfClass(UITextField.self)).atIndex(0).perform(grey_typeText("12345"))

EarlGrey.select(grey_kindOfClass(UITextField.self)).atIndex(1).perform(grey_typeText("Username"))

EarlGrey.select(grey_buttonTitle("Log in")).perform(grey_tap())

EarlGrey.select(grey_text("Is UI testing easy?")).assert(grey_sufficientlyVisible())

}

29

Under the hood

Application state tracking is backed by solid reverseengineering and lots of method swizzling:

• CALayer→ setNeedsDisplay• NSURLConnection→sendAsynchronousRequest:queue:completionHandler:

• UIViewController→ viewWillAppear:• UIAnimation→ markStart:, markStop• Many more...

30

Fast animations

EarlGrey provides a simple API to speed up animations. Result:way faster suite execution.

class FeatureTests: XCTestCase {override func setUp() {super.setUp()GREYTestHelper.enableFastAnimation()

}}

31

Offscreen element matching

Scrolling down the table view.

func testMatchesOffScreenCell() {EarlGrey.select(grey_text("Cell Text"))

.using(grey_scrollInDirection(.down, 50.0),on: grey_kindOfClass(UITableView.self))

.assert(grey_sufficientlyVisible())}

32

Other features

Other interesting EarlGrey features are:

• Changing device orientation.• Layout assertions:

• Verifies conditions like: <ViewA> is to the left of <ViewB>.• Verbose API modelled after NSLayoutConstraint.

• Handles a plethora of interactions: tap, pinch, swipe, etc.• Rendering views to UIImage.

33

Issues

• Application state managed by hand:• EarlGrey test case cannot restart the application.

• No support for 3D-Touch.• EarlGrey cannot interact with system permission dialogs.• Asserting complete view hierarchy is painful.

34

EarlGrey + snapshots = r

Test case

−→vote

35

Implementation

func testPositiveVoteWorks() {EarlGrey.select(grey_kindOfClass(SingleAnswerView.self))

.atIndex(1)

.perform(grey_tap())

.assert(grey_sufficientlyVisible())

EarlGrey.select(grey_kindOfClass(AnswerPicker.self)).assert(grey_verifyDeviceAgnosticSnapshot())

//.assert(grey_recordDeviceAgnosticSnapshot())}

36

EarlGreySnapshots

elpassion/EarlGreySnapshots

Collection of snapshot assertions for EarlGrey. Key benefits:

• Asserting complete view hierarchies.• Visual failure diff instead of cryptic log messages.• Great technique to work with legacy code:

• Freezing a couple of critical paths is possible regardless ofthe code quality.

• Slightly faster feedback loop than without the tests at all.

37

Thank you

Thank you for yourattention!

38

Further reading

Jakub TurekAutomated UI testing for iOS appshttps://jakubturek.com/automated-ui-tests-in-ios-apps/

Orta TheroxSnapshot Testinghttps://www.objc.io/issues/15-testing/snapshot-testing/

EarlGrey TeamGetting started with EarlGreyhttps://github.com/google/EarlGrey/

39