Mobile and Pervasive Computing - 3 Pervasive Computing User Studies Presented by: Dr. Adeel Akram...

Mobile and Pervasive Computing - 3

Pervasive Computing User Studies

Presented by: Dr. Adeel AkramUniversity of Engineering and Technology,

Taxila,Pakistan

http://web.uettaxila.edu.pk/CMS/SP2014/teMPCms

A.J. Bernheim Brush, [email protected]

Outline

Introduction

Types of Studies

Study Design

Example

Ten Mistakes to Avoid

Your Turn

Dr. A.J. Bernheim Brush ?

Ph.D. in Computer Science

Researcher at Microsoft Research

Technology for families, workgroups (HCI/CSCW/Ubicomp)

Researching on various “studies”

http://research.microsoft.com/en-us/groups/CSCW/

Why do a user study?

Bad Reasons

You don’t have anything else to do…

You think it’s a requirement to get your paper accepted

It might be fun to see how people use your stuff

Good Reasons

You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it?

There is a new domain or Scenario and you want to observe user behavior

The Reality….

User Studies are a lot of work Really, more work than you ever expected

Understanding your GOAL is critical

Studying current behavior: What are people doing now?

Proof of concept: Does my novel technology work for people?

Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things

Outline

Introduction

Types of Studies

Study Design

Example


Your Turn

Many types of studiesFo

rmat

ive/

Cur

rent

Beh

avio

r

EthnographySurveyInterviewsFocus GroupsLogging

Pro

toty

ping

/

Pro

of o

f

Con

cept

“Discount Usability”Heuristic EvaluationsLab studies

Learn about a domainDesign inspiration

Typ

es

Goal

Sum

mat

ive/

Usa

ge

Expe

rien

ce

Iterative TestingFix & understand your prototype

Lab studiesField StudiesLogging

Does it work?Can people use it?How does it compare to other designs/prototypes?

Many types of studiesFo

rmat

ive/

Cur

rent

Beh

avio

r

EthnographySurveyInterviewsFocus GroupsLogging

Pro

toty

ping

/

Pro

of o

f

Con

cept

“Discount Usability”Heuristic EvaluationsLab studies

Learn about a domainDesign inspiration

Typ

es

Goal

Iterative TestingFix & understand your prototype

Lab studiesField StudiesLogging

Does it work?Can people use it?How does it compare to other designs/prototypes?

Sum

mat

ive/

Usa

ge

Expe

rien

ce

Surveys

Easy to get large number of people

Design guidance

Evaluation of deployed system

Surprisingly hard to do well….

Phrasing of questions

Biased responses

Pilot your survey!

“Discount Usability” (Jakob Nielsen)

If you are building prototype, very useful to get feedback from users early and often

Low-cost, Quick, Iterative, Small N, Identify big problems

Lo-fi prototypes Paper version can be very helpful

People feel ok telling you to change stuff

No feedback on responsiveness etc.

Heuristic evaluation Experts review the interface based on list of heuristics

Cognitive walk-through Determine tasks, review and ask questions for each task

WARNING: Not typically a research contribution

http://www.nngroup.com/people/jakob-nielsen/

Lab Studies

Bring participants into a lab

Minimize variability

Hypothesis testing

Independent variables between conditions

Interface A vs. Interface B

Control condition?

Measure dependent variable

Speed of use, … (Quantitative)

Preference, …. (Qualitative)

http://www2.sta.uwi.edu/usability/facilities.htm

Field study

In-situ (“not on your turf”)

Trading “control” for realism

Think carefully if this is important

All types

current behavior,

proof-of-concept

prototype

WARNING: Often good idea to do lab study before field study

How do I choose?

You might not…

What is your goal/research contribution?

LINC: An Inkable Digital Calendar

http://research.microsoft.com/pubs/69395/cscw2006lincvideo.pdf http://research.microsoft.com/en-us/um/people/ajbrush/LINCCSCW2006Video.wmv

Research Question/Goal

Your Research Question is critical

There are very few right decisions, instead decisions you need to justify

Bad: How will families use SPARCS?Better: Do sharing suggestions promote sharing?

http://research.microsoft.com/apps/pubs/default.aspx?id=73932

Outline

Introduction

Types of Studies

Study Design

Example


Your Turn

Study Design

What type of study?

What will your participants do during the study?

Give them hardware? Give them tasks?

What type of participants you should recruit?

What data will you collect?

How long will the study be?

Where can you skimp during the study….

What absolutely has to work (if it’s a prototype)

Human Subjects

Ethical treatment of people in the study

Respect—remember that they are doing YOU a favor

Participants can stop at any time

Consent Forms

Privacy Statements

Compensation

Your organization should have some review process

THIS IS IMPORTANT! (U.S. Department of Health & Human Services)

What will they be doing during the study

How will you report on what they did

http://www.hhs.gov/ohrp/assurances/index.html

PILOT!!!!

Allow time to a pilot study,

Run through the entire methodology with volunteer participants.

Uncover system problems

Uncover experimental design problems

Uncover problems with materials

IMPORTANT IMPORTANT IMPORTANT

Participant Profile

What types people do you want to participate?

All the same?

Participant Profile


All the same?

All different?

Participant Profile


All the same?

All different?

People with extreme characteristics vs. “normal” people

Participant Profile


All the same?

All different?

People with extreme characteristics vs. “normal” people

Consider

Age

Gender

Technology experience

….

How many participants?

This can be a difficult question

What claims you are making

What is feasible

Some will drop out!

Length

How long should the study be?

Another difficult question….

Novelty

What are you asking participant to do?

Long term use vs. feasibility

Data Collection

How will you understand if you answered your research question?

Quantitative Data (logs, timing, # errors, …) Qualitative Data (interviews, surveys, …)

TRIANGULATE between multiple sources

Logging

You must have a plan going in about how you will use the log data

Risk of forgetting to log something important

Logging too much can create an analysis nightmare

Make a list of questions you expect to answer with log data

How many times did they upload a photo?

How many days did they use your prototype?

Logging

Question How/Notes1. How often was SPARCS running, when did it stop? # of logfiles, LogfileStart(0)

LogfileClosed

2. Is automatic previewing setup? When Settings:IsPreviewing

3. Is automatic publishing setup? When Settings:IsAutoPublishing

4. How many people’s feeds were they subscribed to? SPARCS_Feed ADD as well as number of unique Create View

5. When did they add or subtract feeds? SPARCS_Feed, add or remove tags

6. What type of calendar feed are they using Calendar_Type, Settings:CalendarSource tag

Qualitative Data

Surveys

Pre-survey

Post condition

Post-survey

Experience Sampling Methodology

Small set of question

Event triggered, random….

Diaries

Interviews (semi-structured, structured)

Observation

Analyzing Qualitative Data

Affinity Diagramming

Coding of Comments

Inter-rater reliability

What if it doesn’t work?

There are many ways a study can fail

Technical problems

They don’t like it

Nobody but you cares about usability problems

Brace yourself for this

Figure out what has to work and skimp other places

Comparison between prototypes

Pilot studies

Outline

Introduction

Types of Studies

Study Design

Example


Your Turn

Example

CareNet Consolvo, Roessler,

Selton Intel Research Seattle

The CareNet Display: Lessons Learned from an In

Home Evaluation of an Ambient Display

Sunny Consolvo, Peter Roessler, & Brett E. Shelton

Intel Research Seattle November ’04

November '04 The CareNet Display * Intel Research Seattle 34

Meet Rita…

Rita is one of the elders we worked with 83 years old Lives alone Conditions: Mild dementia & Type 2 Diabetes

(takes insulin)

She is struggling to maintain her independence, but needs help…

November '04

The CareNet Display * Intel Research Seattle 35

Rita’s Care Network

Drastic life changer:daughter Hannah

Significant contributor:son Simon

Significant contributor:son Zack

Neighbor

Daughter-in-law

Part-time professional caregiver

Son

Hannah’s boyfriend

Daughter-in-law

November '04


The focus of this work

Improve the quality of life for ALL care network members, including the elder

Help members coordinate care activities

Ensure elder gets care she needs

Give time back to overburdened members


The CareNet Display

Interactive digital picture frame always-on Internet access from

wireless GPRS card

Augments photo with updates meds, meals, outings, activities,

mood, falls, & shared calendar

Goal: help local care network members provide daily care

November '04


Participants & Methodology

13 participants from 4 care networks In home Wizard of Oz deployments

i.e., phone calls instead of sensors

Interviews & surveys No special instructions on:

how or when to use it, or where to place it

November '04


Some Results

Drastic Life Changers:

time back, less stress, “meaningful” conversations

Significant Contributors:

increased awareness of what others contribute

Peripherally Involved Members:

gave them something to talk about

Elders:

changed minds about how open they would be


Challenges: Sensor data with a “human touch” Participants are afraid sensor data will be too impersonal How can we provide this “human touch” without adding to the

responsibilities of already overburdened network members?

November '04


Early & in situ evaluations

Early evaluations help us learn where to concentrate our technology development

In situ evaluations provide us with valuable insight into how new tools are used and what effects they have on the intended users

November '04


For more information…

Publications: S. Consolvo, P. Roessler, & B.E. Shelton, "The

CareNet Display: Lessons Learned from an In Home Evaluation of an Ambient Display," Proceedings of the 6th Int'l Conference on Ubiquitous Computing: UbiComp '04 (Sep 2004), pp.1-17.

S. Consolvo, P. Roessler, B.E. Shelton, A. LaMarca, B. Schilit, & S. Bly, "Technology for Care Networks of Elders," IEEE Pervasive Computing Mobile and Ubiquitous Systems: Successful Aging, Vol. 3, No. 2, (Apr-Jun 2004), pp.22-29.

Lots of other examples…

SuperBreak Morris, Brush, Meyers

Microsoft Research

ButterflyNet Yeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger,

Paepcke Stanford, Maryland

Wearable Jersey DisplayPage and Moere

University of Sydney

UbiFitConsolvo, McDonald, Toscos,

Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand,

Libby, Smith, LandayIntel Research Seattle

Outline

Introduction

Types of Studies

Study Design

Example


Your Turn

#10 Not enough people involved

It takes a village

Huge time commitment, 24 hour support for field studies

Do

Include multiple people

#9 Not being prepared

Don’t want to realize at the end of the study that you forgot to do something important

Do

Study design document

Research Question

Participant Profile

Methodology (within/between etc)

Timeline

Data collection

Pilot studies

WARNING: If you do not plan, you plan to fail!

#8 Not enough time for logistics

Everything takes time…. (more than you think)

Recruiting

Installation (e.g. 16 people X 2 hours = )

Support

Do

Allow plenty of time for the study

Have enough people

#7 Seeing what you want to see

We all want our prototypes to be popular

Do:

Think carefully about how you discuss the technology with participants

Avoid leading questions

Stay close to the data and find multiple support for conclusions

Use neutral language, “Tell me more,” “ummm”

Don’t get defensive

#6 Being judgmental

Users are always “right”

They may say things that are offensive, objectionable, etc.

Do:

Leave your opinions at home

Collect feedback

Use neutral language

#5 Not monitoring usage

Don’t want to find out at the end of the study that people were not using the prototype

Do:

“phone home” messages

Server logs

Have a plan about when and how you might intervene

#4 Not collecting a variety of data

Hard to understand logs without interviews

Hard to know if what someone says in an interview matches their use without logs

Do

Interviews

Surveys

Logs

Support for your findings in multiple ways gives you (and reviewers) more confidence

#3 Prototype is not robust enough

Technical challenges are a distraction

You are making a “product”

Do

Usability studies

Pilots with friendly folks

#2 Making inappropriate claims

One of the biggest and most common mistakes

Your study is one data point

Do:

Being careful with your language

Include a “limitations” section

Tell a clear story (don’t have to tell every finding)

#1 Not having a clear research question

Hard to explain the choices you made

Hard to explain your findings

May end up focusing on usability problems

Do

Have a clear research question

Acknowledgments

Sunny ConsolvoIntel Research

Seattle

Beverly Harrison Intel Research

Seattle

Ed CutrellMSR India

User Studies Are Fun!

Always learn something Often surprising! Inspire you

Questions???

References

Brush, Ubiquitous Computing Field Studies in Ubiquitous Computing Fundamentals. A.J. Brush, [email protected]

Green & Salkind Using SPSS for the Windows and Macintosh: Analyzing and Understanding Data (nth Ed.)

APA Publication Manual http://www.apastyle.org/manual/index.aspx

HCI textbooks Preece, J., Rogers, Y., and Sharp, H. Interaction design: beyond

human-computer interaction. 2002. John Wiley & Sons, Inc

Mobile and Pervasive Computing - 3 Pervasive Computing User Studies Presented by: Dr. Adeel Akram...

Documents

Transcript of Mobile and Pervasive Computing - 3 Pervasive Computing User Studies Presented by: Dr. Adeel Akram...