Bernard, packet core engineer at NSN

Belén, interaction designer at Intel’s OTC

how to plan and run usability tests with mobile devices in a laboratory setting

assuming ...

1. you have experience planning and running usability tests with desktop computers

2. you are testing software or an interactive prototype running on a mobile device

3. you are testing in a usability laboratory or similar environment

we will not focus on

1. the fundamentals of the usability testing technique

2. how to prototype for mobile devices

3. how to test in the field

these are the golden rules you need to remember

let us begin ...

usability testinga process that employs people as testing participants who are representative of the target audience to evaluate the degree to which a product meets specific usability criteria.

Handbook of usability testing 2nd Ed., J. Rubin and D. Chisnell

usability testing

watching people try to use what you’re creating/designing/building (...) with the intention of (a)making it easier for people to use or (b)proving that it is easy to use.

Rocket Surgery Made Easy, S. Krug

dut vs mut

Product Desktop software

Mobilesoftware

Device Computer Mobile phone

Context Predictable Unpredictable

Connection DSL / Cable Mobile phone network

which phone?

which context?

which connection?

which phone?

which context?

which connection?

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

web task success rates

feature phones 38%

smartphones 55%

touch phones 75%

Mobile usability, J. Nielsen’s Alertbox 20 Jul 2009http://www.useit.com/alertbox/mobile-usability-study-1.html

handset usability affects test results

test with participants’ own phones

when not possible, include training and

warm-up tasks

handset becomes a screening criteria when recruiting test subjects

http://goo.gl/hmp8t

mobile phone subscriptions per 100 people

http://data.worldbank.org

http://gs.statcounter.com/

http://gs.statcounter.com/

WP iOS Android Blackberry

http://developer.android.com/resources/dashboard/platform-versions.html

http://us.blackberry.com/developers/choosingtargetos.jsp

do not confuse target platform with target user

which phone would you use for testing?

who are the target users?

handset screening criteria

will you need warm-up tasks? (yes / no)

if yes, which ones?

BNT are considering a mobile version of their website.

Before investing in the project, they would like to know how their existing desktop site performs on mobile devices.

You are working on an Android application for Sofia Airport.

Together with your developers, you have created a first version of the app. This ‘alpha’ version is not available in Google Play, it is not fully tested and might be buggy.

St. Ekaterina University Hospital has developed a Blackberry application that helps nurses and doctors access and update patients records during their daily shift.

They have asked you to help them test the application.

Evernote wants to uncover the main usability issues in their existing mobile applications for iOS, Android, Windows Phone and Blackberry.

Help them organise the necessary testing.

what about emulators?

mas_fsx_ by Massimiliano Silipigni at http://www.flickr.com/photos/silipigni/6055803785/

good for ...

using an emulator enables testers to thoroughly capture user behaviour (...) The captured data are generally informative and useful for analysing user performance

Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)

good for ...

emulators are more suitable to be used for improving the interface design of applications such as the layout of menu structures during the development process.

Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)

but bad for ...

using emulators alleviates the problems of latency, inefficient input mechanisms, and the changing wireless environment, potentially leading to untruthful user satisfaction

Challenges, Methodologies, and Issues in the Usability Testing of Mobile Applications, D. Zhang, B. Adipat, International Journal of Human Computer Studies (2005)

which phone?

which context?

which connection?

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

field vs. lab

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

field vs. lab

0 0

The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.

Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

The results show that the added value of conducting usability evaluations in the field is very little and that recreating central aspects of the use context in a laboratory setting enables the identification of the same usability problem list.

Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the FieldJ. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

0 1Field Lab

according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported

Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

according to our study there was no difference in the number of problems that occurred in the two test settings. Our hypothesis that more problems would be found in the field was not supported

Usability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

0 2Field Lab

evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

evaluations conducted in field settings can reveal problems not otherwise identified in laboratory evaluations

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

1 2Field Lab

The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.

Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

The analyses of the comparison between usability testing done in two different settings revealed that there were many more types and occurrences of usability problems found in the field than in the laboratory. Those problems discovered tend to be critical issues.

Usability Evaluation of Mobile Device: a Comparison of Laboratory and Field TestsH.B Duh, G. C. B. Tan, V. H. Chen, MobileHCI 2006

2 2Field Lab

field vs. lab

EXPERTS DISAGREE

It’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

... but they all agree

testing in the field requires double the time in comparison to the laboratoryUsability Testing of Mobile Applications: A Comparison between Laboratory and Field TestingA. Kaikkonen, T. Kallio, A. Kekäläinen, A. Kankainen, M. Cankar - Journal of Usability Studies, 2005

... but they all agree

evaluations in the field (are) more complex and time-consumingIt’s Worth the Hassle! The Added Value of Evaluating the Usability of Mobile Systems in the FieldC.M. Nielsen, M. Overgaard, M.B. Pedersen, J. Stage, S. Stenild - NordiCHI 2006

... but they all agree

field-based usability studies are not easy to conduct. They are time consuming and the added value is questionable.Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M. B. Skov, B. S. Als, R. T. Høegh, 2004

testing in the lab is better than no testing

can you think of an example of mobile software that you must test in the field?

do it late

plan an run pilot tests

be prepared (like the Scouts)

if doing field testing ...

have I chosen an appropriate location?

which external factors could impact results?

is the equipment working as planned?

do I have enough time to run the tests?

does the protocol work? Is it well understood?

if doing field testing ...

If we cannot go to the field, the field must come to us

Simulations

Scenarios

Take the lab out

Bringing the field into the lab

simulations

Use of techniques to “reflect or recreate a mobile use situation” (...) their level of sophistication is measured by the extent to which the environmental and ergonomic factors of the actual use context are recreated.

Emerging Research Methods for Understanding Mobile Technology Use, P. Hagen, T. Robertson, M. Kan and K. Sadler, Proceedings of OZCHI 2005, Canberra (Australia)

Is it Worth the Hassle? Exploring the Added Value of Evaluating the Usability of Context-Aware Mobile Systems in the Field, J. Kjeldskov, M.B. Skov, B.S. Als and R.T.Høegh, MobileHCI 2004

Experimental Evaluation of Techniques for Usability Testing of Mobile Systems in a Laboratory Setting, E.T. Beck, M.K. Christiansen, J. Kjedskov, N. Kolbe and J. Stage, OZCHI 2003, Brisbane (Australia)

scenarios

First you choose the tasks to test (...) Then you expand these tasks into scenarios (...) A scenario provides some context and supplies information the user needs to know

Rocket Surgery Made Easy, S. Krug

a task:Find a hotel

a desktop scenario:You are planning a surprise weekend trip to Paris. You’ll be leaving on Friday and coming back on Sunday evening. Your partner’s favourite Paris area is Le Marais. Find a place to stay he/she will like.

a task:Find a hotel

a mobile scenario:You were late to Orly airport and have missed your flight home. There are no more flights from Paris today, so you’ll have to spend the night. Since it’s already 10 pm, use your phone to look for a place to stay close to the airport.

portable labs can be deployed anywhere

document camera

laptop

webcam

Jyske Bank in Roskilde by king_david_uk at http://www.flickr.com/photos/kingdavid/2926673466/

Vodafone, Croydon, London CR0 by Kate Pugh at http://www.flickr.com/photos/kake_pugh/6545344321/

write a mobile scenario for your

group’s task

You are testing:

http://www.tfl.gov.uk

Task:

Find a public transport route

A bit of context:

From Heathrow Terminal 5 to the Olympic Stadium. User must arrive on time for the Olympics opening ceremony (27th July at 7:30pm)

You are testing:

http://www.toptable.co.uk

Task:

Find a restaurant

A bit of context:

Lebanese cuisine Around Shoreditch (an East London area)For tomorrow at 8:30pm For 5 people

You are testing:

An Android app called ‘CycleStreets’, which helps you find cycling routes across London

Task:

Calculate a route distance

A bit of context:

From King’s Cross station to Tate ModernRoute with least traffic

You are testing:

A task management app called Remember the Milk

Task:

Create a task

A bit of context:

The task is to buy flowers for a friend (make up a name)Next Tuesday morningIn her favourite flower shop, called Blooming

which phone?

which context?

which connection?

how often are mobile users connecting their devices to WiFi networks instead of mobile networks?

Towering by Rachel Strohm at http://www.flickr.com/photos/rachelstrohm/4693016049/

25% of households worldwide have WiFi networks set up

Globally, smartphones account for 36% of all public WiFi hotspot connections, compared to 48% for laptops and 10% for tablets

Out of 4 billion unique mobile phone users, 1 billion uses WiFi to access the Internet (there are only about 900 million smartphones)

http://www.lukew.com/ff/entry.asp?1537

http://communities-dominate.blogs.com/brands/2012/05/mobile-services-beyond-messaging-excellent-tns-global-survey-reveals-tons.html

http://databank.worldbank.org

do not test over WiFi

cover your participants’ data costs

and on the big day ...

remote?

Remote Mania by Dramagirl at http://www.flickr.com/photos/dramagirl/2268235917/

http://www.usertesting.com/mobile

but ...I’d say remote testing gives you about 80% of the benefits of a live test with about 70% of the effort. You do lose 20% of something not being in the same room as the participant. The in-person experience is just richer, somehow.

Rocket Surgery Made Easy, S. Krug

To be or not to be, that is the question by Mister_Jack at http://www.flickr.com/photos/machineman22/5130901135/

to record or not to record ...

it is difficult ...We have used Morae in the past also for website testing, but could not easily use the system for mobile (...) Obviously it’s difficult to video something like that so we just sit next to them and observe.

http://belenpena.posterous.com/user-research-for-mobile-software-a-short-sur

... but video is both a

memory aid

powerful communication tool

reactions

actions

actions

reactions

which phone?

which context?

which connection?

which recording setup?

Mobiola Screen Capture for Blackberry 4.2+ and Symbian S60 v3http://www.shapeservices.com/en/products/details.php?product=capture&platform=none#

document camera records mobile screen

USB webcam records participant’s face

both cameras connect to a laptop running screen

recording software

document camera range is marked with tape

Mobile Device Camera by Noldus http://www.noldus.com/human-behavior-research/accessories/mobile-device-camera-mdc#

Ovo Studios device camerahttp://www.ovostudios.com/devicecamera.asp

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

Nick Bowmast http://www.bowmast.com/mob-device-cam/

Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

Little Springs Designhttp://www.littlespringsdesign.com/blog/2008/Jun/usability-testing-for-mobile-devices-2/

Nick Bowmast http://www.bowmast.com/mob-device-cam/

Usability Scienceshttp://www.usabilitysciences.com/services/lab-based-usability-testing/mobile-usability-testing

Google Towards the Perfect Infrastructure for Usability Testing on Mobile Devices, R. Schusteritsch, C.Y. Wei, M. LaRosa - Google (CHI 2007)

are blooming!!

screen audio

Skype Skype

Screen sharingTelephone conferencing system

Mirroring

Telephone conferencing system

the observation room

Screen sharing

Mirroring

Telephone conferencing system

pros cons

provides high quality screen recording

participants won’t appreciate you installing stuff on their phones

provides high quality screen recording no application will

support all platforms

provides high quality screen recording

in touchscreen phones, finger interactions are not captured

screen capture applications

Ovo Studios screen capture application for iOShttp://www.ovostudios.com/

Obviously,

think aloud is critical because I cannot see how the participant is interacting

with his fingers on the touch screen.

I can see using the mirroring in a lab setting to get the signal from the iPad

to the observation room though you still won’t see the physical interaction

with the device, like you would with a device mounted camera (e.g. Noldus).

... and that is a big deal

Why AirPlay mirroring is the Biggest Thing to Happen to User Research in 2011 - http://www.remoteresear.ch/airplay/

Recently, we’ve questioned the value of capturing the device screen. It does

after all end up being a video of the screen changing but with no sense

of the participant interacting with it.

iPad usability testing - our equipment - http://www.cxpartners.co.uk/cxblog/ipad_usability_testing_-_our_equipment/

... and that is a big deal

pros cons

good recording quality document cameras bring up the price

easy to set up participants must keep within the camera range

phone must lay on a desk or be hold at a flat angle

document cameras

pros cons

allow natural interaction with the phone

ready-made devices will bring up the price

allow natural interaction with the phone

if bulky, they can prevent single-hand use

allow natural interaction with the phone

if heavy, they can become tiring during long tests

mounted devices

the ideal recording setup

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face & fingers

gives enough video quality

the ideal recording setup screen capture

applications

document cameras

mounted devices

ready-made

mounted devices DIY

easy to put together

cheap

repeatable

allows holding the device

allows one-handed use

supports all form factors

runs tests with participants’ phones

captures screen, face & fingers

gives enough video quality

no recording setup is perfect: just use the best tool for the job

?

test with participants’ own phones

do not confuse target platform with target user

testing in the lab is better than no testing

do not test over WiFi

cover your participant’s data costs

no recording setup is perfect: just use the best tool for the job

thanks!!!

www.DIYMobileUsabilityTesting.net