Remote Mobile User Testing Workshop (UXDC 2017)

Workshop:Remote Mobile User TestingNew Tools for Moderated Mobile Testing at a DistanceTristan Wilson • Dana Douglas

www.userworks.com@userworks

Workshop: Remote Mobile Testing2

Research ScenarioRemote mobile story time

Workshop: Remote Mobile Testing 3

Research Scenario

Email from a Client


Research Scenario

Let's break it down. What stands out?

Hey Tristan,We have a new mobile app that the team is working on that we’d like you to test. It’s a functional prototype, so users should be able to click around and see some of the functionality. We want to test it with a geographic mix of smart phone users in the U.S. – let’s say 8 participants. We’d like to learn whether people can navigate and understand the labels, as well as just get their general impressions of the app.Our design team in San Francisco would like to observe the sessions live – is that ok?As always, the budget’s pretty tight, so let’s keep this as inexpensive as possible.Can you work up a plan to present to the team next week?Thanks,Dana


Research Scenario

Major takeaways from email• Testing for a new mobile app• Functional prototype• Want to test user interaction• 8 participants desired • Need a geographic mix of participants• Want to observe sessions remotely• Budget is "pretty tight"

What will our study look like?


Remote TestingA Bit of Background


What We Mean When We Say...

Remote TestingParticipant and researcher in different geographical locationsSynchronous research, live-moderated

Mobile Modern, portable “smart” devices Most commonly smartphones and tablets


Benefits of Remote Testing?

More naturalistic environment

Lower cost & increased efficiency

Greater convenience

Expanded recruitment capability


Drawbacks of Remote Testing?

Potential reduction in quality of data

Expanded spectrum of technical issues

Reduced scope of research

Diminished participant-researcher interaction


Drawbacks Specific to Mobile?

Range of OS versions across devices

Lack of mobile interaction tracking/capturing Touch, gestures, etc.

Participant currently has to share their own screen Creates the potential for privacy concerns

Wireless can lead to reduced speed & reliability for data transfer


Existing MethodsYesterday's Remote Mobile


The Browser ResizeConnect to researcher’s computer remotelyInteract with browser sized to mobile dimensionsUse laptop/desktop PC, – not mobile device

° Researcher resizes browser on their own computer to match dimensions/pixel density mobile device

° Testing conducted like traditional remote session on PC, but mimicking mobile

° Participant interacts with the mobile-sized browser remotely with mouse/keyboard


The Standalone Webcam

° Participant uses an external webcam to take video of themself as they interact with mobile device

° Video feed is live-streamed to researchers located remotely

Not an accurate depiction

Position external webcam to capture deviceStream interactions to remote researchers


The “Laptop Hug”Wrap arms around reverse-facing laptopInteract with device in view of laptop webcam

° Like standalone webcam method, participants stream a video feed of their mobile interactions

° Instead of a separate webcam, participants use their laptop (or tablet) to take video

° Hinged laptop facilitates camera angle; easier on participant


New OptionsTomorrow's Remote Mobile


Ingredients for New Remote Mobile


Mobile Mirroring

Commonly via native wireless “casting” protocols:

Google Cast(Android)

AirPlay(iOS)


Mobile Streaming

Directly streaming from the mobile device itself- or -

Re-capturing & streaming a locally-mirrored screen using a PC


Setup A


Setup B


Setup C


Setup D

23

Remote Mobile Setup Types

PARTICIPANT RESEARCHER


Dream ToolCharacteristics of an Ideal Solution


Characteristics of the Ideal Tool

Revisiting the client's email: • Testing for a new mobile app• Functional prototype• Want to test user interaction• 8 participants desired • Need a geographic mix of participants• Want to observe sessions remotely• Budget is "pretty tight"


Characteristics of the Ideal Tool

What characteristics would you be looking for in a software tool that can help you

meet your client’s needs?


Characteristics of the Ideal Tool° Ease of Use

° Painless for participants to set up and use; not intimidating° Quick and simple to initiate screen sharing from both moderator and participant

perspective° Allows for remote mobile mirroring without a local computer as an intermediary° Easy for moderator to implement and control

° Price & Affordability° The cheaper the better if all else is comparable


Characteristics of the Ideal Tool° Performance & Reliability

° Minimal lag time° Can run alongside other applications without impacting experience/performance° Accurate representation of participants' actions and gestures

° Range of Features° Ability to carry out other vital aspects of research in addition to screen sharing

(e.g., web conferencing, in-app communication, recording)° Platform agnostic: fully functional on all major mobile platforms, particularly iOS

and Android. ° Allows participants to make calls while mirroring screen


Characteristics of the Ideal Tool° Privacy & Security

° Allow participants to remotely control researcher’s mobile device via their own

OR, if participant must share their own screen:

° Considers participant privacy by clearly warning when mirroring begins° Provides participant with complete control over start and stop of screen sharing° Snoozes device notifications while sharing screen° Shares only one application rather than mirroring the whole screen


DemonstrationMobizen


MobizenA relatively simple mirroring tool.

Platform: Android, iOS

Price: Free (Android Only) $3.99/mo (Enables iOS)

Features: - Mirroring- Streaming- Recording- Annotation


Mobizen

What setup type is Mobizen?

iOS:

Android:


Android Demonstration

1. Moderator installs Mobizen on computer2. Participant installs Mobizen on mobile device3. Sign in with same account on both devices4. Mod select mobile device from computer and connects5. Participant accepts connection on mobile device


Discussion

How would you rate Mobizen?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features

ParticipantEase

ModeratorEase

Price &Affordability

Performance & Reliability

Range ofFeatures

1

10


Break


Research Logistics


Research Logistics

Participant Recruiting


Research Logistics

Participant Privacy and Security


Research Logistics

Recording & Observers


Research Logistics

Technical Setup with Participants


Research Logistics

Running the Session


Lessons We've Learned

Planning ahead is key

Practice makes perfect

Always have a backup

Put participants at ease

Tailor recruitment to make your life easier


Group Demos


Group Activity° Learn about a tool with your group° Test out the tool with a mobile device° Think about this process in the context of a remote usability test° Rate the tool on the following:

° Participant Ease of Use° Moderator Convenience° Affordability° Performance & Reliability° Range of Features

° Present your tool to the rest of us


ParticipantEase

ModeratorEase



Range ofFeatures

1

10

Group Recap: Zoom

How did you rate Zoom?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features


ParticipantEase

ModeratorEase



Range ofFeatures

1

10

Group Recap: Join.me

How did you rate Join.me?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features


ParticipantEase

ModeratorEase



Range ofFeatures

1

10

Group Recap: Reflector 2

How did you rate Reflector 2?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features


ParticipantEase

ModeratorEase



Range ofFeatures

1

10

Group Recap: AirServer

How did you rate Airserver?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features


ParticipantEase

ModeratorEase



Range ofFeatures

1

10

Group Recap: TeamViewer

How did you rate TeamViewer?° Participant Ease of Use° Moderator Ease of Use° Price & Affordability° Performance & Reliability° Range of Features


What Tool Would You Choose?Back to the email: • Testing for a new mobile app• Functional prototype• Want to test user interaction• 8 participants desired • Need a geographic mix of participants• Want to observe sessions remotely• Budget is "pretty tight"


What Tool Would You Choose?

What tool would you propose to use in response to your client’s request?


The FutureWhat Comes Next


The Future: Remote Mobile ToolsIdeal Tool Dimensions

Ease of UsePrice & Affordability

Performance & ReliabilityRange of FeaturesPrivacy & Security


The Future: Remote Mobile ToolsSpecific areas for improvementFeatures

° Ability to work reliably with Android AND iOS

Privacy° Screen sharing of one particular app rather than the whole screen° Sharing control of the researcher’s mobile device screen

55Workshop: Remote Mobile Testing

Tristan WilsonUser Experience Specialist

[email protected]@tristanwilson

Dana DouglasUser Experience Specialist

[email protected]


Appendix: Workshop Handouts

Remote Mobile User Testing Workshop (UXDC 2017)

Technology

Transcript of Remote Mobile User Testing Workshop (UXDC 2017)