TEST MANAGERS FORUM 2014 SYDNEY, AUSTRALIA MOBILE TESTING WORKSHOP

1

2

WHO AM I? Gridfusion Software Solutions Contact: Michael Palotas Gerbiweg 2 8853 Lachen SWITZERLAND Tel.: +41 79 6690708 Email: michael.palotas@gridfusion.net

Head of Productivity & Test Engineering, eBay

Founder / Principal Consultant Gridfusion Software Solutions

SETTING THE STAGE

Tell me about yourself J

What are your expectations for today?

3

MOBILE TESTING SPECIFICALITIES

4

MOBILE IS A VAST LAND

5

YOU ONLY HAVE ONE SHOT

6

MOBILE TESTING CHALLENGES DEVICE DIVERSITY

7

MOBILE TESTING CHALLENGES NEW OS VERSIONS

New versions usually fix issues

but

May also introduce new ones

8

MOBILE TESTING CHALLENGES NEW OS VERSIONS

9

MOBILE TESTING CHALLENGES NEW OS VERSIONS

10

NATIVE APPS AND MOBILE WEB

11

MOBILE TESTING CHALLENGES LOCALIZATION

12

MOBILE TESTING CHALLENGES NETWORK

13

MOBILE TESTING CHALLENGES OPERATORS

14

MOBILE TESTING CHALLENGES MOBILE ENVIRONMENT

15

16

17

18

19

MOBILE TESTING CHALLENGES DEVICE MANAGEMENT

20

DEVICE OR EMULATOR?

21

RISK ASSESSMENT AND STRATEGIES

22

RISK & STRATEGIES

Hotfixes are difficult and expensive

Agile?

...yes...but...for mobile the goal should be to get it right the first time

23

RISK & STRATEGIES

May shift back to waterfall

(waterfall is not that bad)

Difficult to get (mobile) experienced people

Need to grow capabilities in-house

24

RISK & STRATEGIES

No automation in most places today

No continous integration / continous delivery

25

RISK & STRATEGIES DEVICE SELECTION

26

Be pragmatic

Focus on where your money is coming from

Forget the “Long Tail”

RISK & STRATEGIES

Multi code base

Many devices

Device / OS management

Simulator vs. real devices

27

CLOUD TESTING

28

CLOUD VENDORS - DEVICE ANYWHERE

29

CLOUD VENDORS - SAUCELABS

30

TESTDROID

31

GENERAL EXPERIENCES WITH THE CLOUD

No dedicated devices

Or no devices at all

Device management

Your company firewall...

Trust issues

Slow(er)

No in-house knowledge

Not onsite

It is a bit like outsourcing

$ in the beginning

$$$$$$ when you are scaling

32

CROWD TESTING

33

REALLY?

34

CROWD TESTING – WHAT USUALLY WORKS?

Fresh set of eyes

Access to many regions, devices, OS combinations

(at least theoretically)

Lots of issues reported

Relatively low cost

35

CROWD TESTING – CHALLENGES

Significant ramp up required for testers

Low bug quality

You don’t know who is on the other end...

People in mature markets won’t work for 5$ / bug

Only access to production environment

QA system access very difficult

Legal implications, NDAs etc.

No development resources to fix the bugs

36

MOBILE DEVELOPMENT APPROACHES

37

MOBILE DEVELOPMENT APPROACHES

Mobile First

Responsive Design

38

TEAM ORGANIZATION

39

AGILE AND MOBILE

CI – OK

CD – OK for QA versions

CD – NOK for prod versions

40

SKILLSET - THE PERFECT TESTER

41

GENERAL TEST AUTOMATION

42

FLIP THE AUTOMATION TRIANGLE

Unit Tests

E2E Automation Tests

Manual Tests

Unit Tests

E2E Tests

Manual Tests

Integration Tests

TEST AUTOMATION

WHY?

44

TEST AUTOMATION

WHO?

45

TEST AUTOMATION

WHAT?

46

TEST AUTOMATION

WHEN?

47

INSTANT FEEDBACK

D T D T D T D T D T D T D T D T

Development Test

AUTOMATION IS GOOD…BUT

Tools help, but tools don’t test

Manual testing is equally important

Manual (exploratory) and automated testing complement each other

49

Automation

Manual

WHY IS E2E TESTING SO IMPORTANT?

Because it is the most difficult way to test

Because our users use our products End2End

Because you can verify that the whole system works together

50

PERSONAL THOUGHTS ON AUTOMATION

Manual testing is still very important

E2E tests are expensive, so keep them to the minimum Automation is software development

Be careful with record & playback tools

Automate the repetitive tasks (regression)

Build testability into your product

ROI...

More at http://gridfusion.net/thoughtsonautomation.html

51

MOBILE TEST AUTOMATION

52

MOBILE TEST AUTOMATION

Not as mature as web automation

Immature tools market

Tools are usually platform specific

Multi code base

Need to understand the underlying architecture much more than for the web

Need people who can code well

53

EMULATORS VS. DEVICES

What to test where?

54

WHAT TO TEST WHERE - EMULATOR

functional test

55

WHAT TO TEST WHERE - DEVICE

performance look and feel

memory usage carrier specific tests

56

IOS SIMULATOR SPECIFICS

57

Comes from Apple

Works well & reliable

ANDROID EMULATOR SPECIFIC

58

SLOW

ANDROID VS. IOS

Android: lots of variations, versions, vendors

Android: open source (sort of)

iOS: easy to update

iOS: everything comes from one vendor

iOS: not exactly open source J

59

SAMPLE MOBILE AUTOMATION REQUIREMENTS

Use the same approach and infrastructure as web

Business is borderless à test automation needs to be as well

Aim for REAL E2E test automation

Use the Load Balancer / Resource Management Features of Selenium Grid

Manage multiple applications / versions / languages

No modification of mobile app

No modification of mobile device

Mobile App Runtime Inspection Support

Hybrid app support

Support for emulator + real device

Integration into CI system

60

SAME AUTOMATION CODE BETWEEN PLATFORMS?

Sounds good first

but

Most apps are different between platforms

Different element locator strategy

Do reuse the helper functions

61

MOBILE AUTOMATION IOS / ANDROID DIFFERENCES

iOS has no IDs à harder to automate

iOS relies on xcode

Android is open source

Android: ID support

62

TEST INFRASTRUCTURE

63

Driver Browsers Mobiles

AUT

DB

API

TOOLSETS - OVERVIEW

Ranorex

QTP

TOSCA

SeeTest

Selenium/ ios-driver/Selendroid

Selenium / APPIUM

64

SeeTest

65

WHAT ABOUT BDD / ATDD?

I have seen it working

Required to close the loop

Removed complexity

(while adding it at the same time)

Have you ever seen a product owner test?

66

SAMPLE TOOLS

JBehave

Fitnesse

Cucumber

67

SELENIUM

68

SELENIUM – WHAT IS IT?

Web testing tool Highly successful open source project Stable

Protocol describing the user interactions

Supports most browser implementations Support for most common programming languages

69

SELENIUM – WHAT IT IS NOT

a drag & drop tool

a network testing / monitoring tool

a performance testing tool

a reporting tool

a testcase management tool

70

SELENIUM

i.e. Firefox driver

Server

JSON WIRE PROTOCOL

Client

Java

C#

Ruby

Python

Server

i.e. Selendroid, iOS-Driver

SCALING - SELENIUM GRID

72

SELENIUM GRID

73

SCALING USING SELENIUM GRID

74

Client

server GRID Hub Client

CI

server

server

server

server

server

server

75

Implements the JSON Wire Protocol (so it is fully compatible with Selenium)

Runs on emulators and devices

Can be run as a node in Selenium Grid

Support for localized native apps (1 test for all locales)

No need to change the AUT

No need to jailbreak the device

76

Hybrid support Different locator types are supported to find elements Gestures are supported: Advanced User Interactions API Hot plugging It can be integrated as a node into the Selenium Grid for scaling and parallel testing Android target API support from 2.3.3 to 4.3 (latest) Built in Inspector to simplify test case development.

REPORTING

Invest into proper reporting Easy analysis of the reports is crucial

Selenium does no reporting

TestNG or JUnit are responsible for reporting 100% customizable reports (i.e. different reporting levels,

PDF, HTML, Email, SMS etc...) Hooks into test case management tools

77

REPORT WITH SCREENSHOT

78

SELENIUM SELENDROID IOS-DRIVER

79