Remote (Mobile) Usability Testing Lab

Remote (Mobile) Usability Testing Lab

Human.Interaction.Design.2010

Jan Moons

[email protected]

November 23rd, 2010

2 | Presentation Title | Month Year Copyright © XXXX Alcatel-Lucent. All rights

reserved.

About

In this presentation we describe our experiences in developing and testing a system that makes it possible to

usability test (mobile) applications.

Disclaimer:Work in progress

Usability Testing?In a nutshell


reserved.

Usability Testing?

A technique to observe people, who are representatives of the target audience, using a product to discover errors and areas of improvement.

Typically measuring how well people respond in following areas:

Efficiency

Effectiveness

Satisfaction


reserved.

5 Characteristics

Every usability test has the following characteristics:

1. Primary goal: Improving usability of a product

2. Participants represent real users

3. Participants perform real tasks

4. Observe and record (log) what participants do and say

5. Analyze the data, diagnose the real problem, and recommend changes to fix those problems

A Practical Guide to Usability Testing, J.S. Dumas and J.C.Redish

Usability Testing @ Bell LabsProblems and Vision


reserved.

Typical usability test @ Bell Labs

Participant

Observer

Logging software

Participants are inevitably invited to the

“lab”!


reserved.

Problems with State-Of-The-Art

1. Software installation needed on participant’s computer

Participants don’t appreciate installing software on their PC’s

Live recruitment is not possible

2. Software doesn’t run on all major operating systems

Target: MS Windows OS (XP/Vista)

Problematic: Mac OS, Linux, Solaris, etc…

3. Observing and recording over the Internet/Intranet hard or even not possible

Users cannot participate from their own “premises” with their own devices

Remote usability testing is an issue

4. Build own Remote Usability Testing system to solve these problems


reserved.

Vision

1. Design a system to remotely observe and record users’ actions, voice and facial expressions on an Internet/Intranet connected PC.

2. With the ability for the observer to analyze the data and diagnose the problems

3. Based on Web technologies that supports all major Operating Systems

4. But without having to install any special software on the participant’s or even the observer’s computer


reserved.

APPAPP

Architecture

Java AppletFlash Applet

RTMP

Audio/Video

Streaming

Screencast

Streaming

RTMP A/V

JDBC

Observer’s browser

Tester’s browser

Remote Objects


reserved.

Observe / Replay

Participant’s webcam

Participant’s screen

Logging panelCurrent task

Mouse pointer


reserved.

Analyze

EffectivenessEfficiency

Data Analysis

Mobile Usability TestingPopular techniques


reserved.

Look-over-shoulder

http://www.wirelessrerc.org/for-consumers

Screen recording?


reserved.

Document Camera

http://www.easeofweb.com/mobile_usability_test.html

Fixed positionUser has to be seated


reserved.

Sled / Lamp

http://www.easeofweb.com/mobile_usability_testing.html

“Mobile” but very intrusive


reserved.

Pros and Cons

Document/Sled Camera:

Pros: Works with all mobile devices No need to install software on mobile Interaction with mobile is visible Ideal for lab testing

Cons: Very intrusive Experience is not natural No automatic key logging


reserved.

Screen-capturingJailbroken iPhone Rooted Android


reserved.

Pros and Cons

Screen-Capturing:

Pros: Less/non-intrusive More natural experience Testing in the field (e.g. Location-Based Services)

Cons: Software installation needed for recording

For each OS (different) implementation needed! Doesn’t show the users hands actually interacting with the mobile

Mobile Usability TestingNatural Experience


reserved.

Vision

1. Extend our custom-built usability testing lab for the computer to mobile devices

2. With the ability to observe and record:

1. The mobile screen / user’s actions

2. The user’s voice (environment noise)

3. The user’s face (optional)

3. Ultimate Goal: Making the experience as natural as possible for user testers


reserved.

Implementation

Prioritized:

1. Camera capturing mobile screen

All mobile devices

Ability to capture interactions (hands)

2. Screen-sharing + Audio/Video streaming

Ultimate goal of a natural experience; non-intrusive

1st focus Android mobile device


reserved.

Approach 1: Camera capturing mobile screen

DIY Document

CameraPC for

transmission of streams to server

Webcam Face

Capturing

Also ideal for testing Paper Prototypes!


reserved.

Architectural Changes

APPAPP

Camera 2

RTMP

Audio/Video

Streaming

Screen

Streaming

RTMP A/V

JDBC


Tester’s browser

Remote Objects

Camera 1


reserved.

Approach 2: A/V Screen-Capturing Screen capturingIssues!!!


reserved.

Observer


reserved.

Architectural Changes

APPAPP

Java AppFlash Applet

RTMP

Video

Streaming

Screencast

Streaming

RTMP A/V

JDBC


Remote Objects

Audio SIP

Tester’s

Mobile

ADB

Screen capture

Connected Computer

Also support for VNC


reserved.

Future work…

1. Measuring satisfaction

Surveys

Emotion recognition (voice, facereading or combination)

2. Automatically generating (video) highlights of observations

3. Capturing the mobile’s sensors:

GPS-coordinates (for Location-Based Services)

Accelerometers, touch events (application / device interaction)

4. SIP/RTMP audio/video streaming

Last but not least, eat our own dog food and usability test the system ;-)


reserved.

www.alcatel-lucent.comwww.alcatel-lucent.com

Remote (Mobile) Usability Testing Lab

Design

Transcript of Remote (Mobile) Usability Testing Lab