Mobile and Pervasive Computing - 3
Pervasive Computing User Studies
Presented by: Dr. Adeel AkramUniversity of Engineering and Technology,
Taxila,Pakistan
http://web.uettaxila.edu.pk/CMS/SP2014/teMPCms
A.J. Bernheim Brush, [email protected]
Outline
Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Dr. A.J. Bernheim Brush ?
Ph.D. in Computer Science
Researcher at Microsoft Research
Technology for families, workgroups (HCI/CSCW/Ubicomp)
Researching on various “studies”
http://research.microsoft.com/en-us/groups/CSCW/
Why do a user study?
Bad Reasons
You don’t have anything else to do…
You think it’s a requirement to get your paper accepted
It might be fun to see how people use your stuff
Good Reasons
You’re designing stuff for people to use. Wouldn’t it be nice to know how they might use it?
There is a new domain or Scenario and you want to observe user behavior
The Reality….
User Studies are a lot of work Really, more work than you ever expected
Understanding your GOAL is critical
Studying current behavior: What are people doing now?
Proof of concept: Does my novel technology work for people?
Experience using a prototype: How does using my prototype change people’s behavior or allow them to do new things
Outline
Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Many types of studiesFo
rmat
ive/
Cur
rent
Beh
avio
r
EthnographySurveyInterviewsFocus GroupsLogging
Pro
toty
ping
/
Pro
of o
f
Con
cept
“Discount Usability”Heuristic EvaluationsLab studies
Learn about a domainDesign inspiration
Typ
es
Goal
Sum
mat
ive/
Usa
ge
Expe
rien
ce
Iterative TestingFix & understand your prototype
Lab studiesField StudiesLogging
Does it work?Can people use it?How does it compare to other designs/prototypes?
Many types of studiesFo
rmat
ive/
Cur
rent
Beh
avio
r
EthnographySurveyInterviewsFocus GroupsLogging
Pro
toty
ping
/
Pro
of o
f
Con
cept
“Discount Usability”Heuristic EvaluationsLab studies
Learn about a domainDesign inspiration
Typ
es
Goal
Iterative TestingFix & understand your prototype
Lab studiesField StudiesLogging
Does it work?Can people use it?How does it compare to other designs/prototypes?
Sum
mat
ive/
Usa
ge
Expe
rien
ce
Surveys
Easy to get large number of people
Design guidance
Evaluation of deployed system
Surprisingly hard to do well….
Phrasing of questions
Biased responses
Pilot your survey!
“Discount Usability” (Jakob Nielsen)
If you are building prototype, very useful to get feedback from users early and often
Low-cost, Quick, Iterative, Small N, Identify big problems
Lo-fi prototypes Paper version can be very helpful
People feel ok telling you to change stuff
No feedback on responsiveness etc.
Heuristic evaluation Experts review the interface based on list of heuristics
Cognitive walk-through Determine tasks, review and ask questions for each task
WARNING: Not typically a research contribution
http://www.nngroup.com/people/jakob-nielsen/
Lab Studies
Bring participants into a lab
Minimize variability
Hypothesis testing
Independent variables between conditions
Interface A vs. Interface B
Control condition?
Measure dependent variable
Speed of use, … (Quantitative)
Preference, …. (Qualitative)
http://www2.sta.uwi.edu/usability/facilities.htm
Field study
In-situ (“not on your turf”)
Trading “control” for realism
Think carefully if this is important
All types
current behavior,
proof-of-concept
prototype
WARNING: Often good idea to do lab study before field study
How do I choose?
You might not…
What is your goal/research contribution?
LINC: An Inkable Digital Calendar
http://research.microsoft.com/pubs/69395/cscw2006lincvideo.pdf http://research.microsoft.com/en-us/um/people/ajbrush/LINCCSCW2006Video.wmv
Research Question/Goal
Your Research Question is critical
There are very few right decisions, instead decisions you need to justify
Bad: How will families use SPARCS?Better: Do sharing suggestions promote sharing?
http://research.microsoft.com/apps/pubs/default.aspx?id=73932
Outline
Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Study Design
What type of study?
What will your participants do during the study?
Give them hardware? Give them tasks?
What type of participants you should recruit?
What data will you collect?
How long will the study be?
Where can you skimp during the study….
What absolutely has to work (if it’s a prototype)
Human Subjects
Ethical treatment of people in the study
Respect—remember that they are doing YOU a favor
Participants can stop at any time
Consent Forms
Privacy Statements
Compensation
Your organization should have some review process
THIS IS IMPORTANT! (U.S. Department of Health & Human Services)
What will they be doing during the study
How will you report on what they did
http://www.hhs.gov/ohrp/assurances/index.html
PILOT!!!!
Allow time to a pilot study,
Run through the entire methodology with volunteer participants.
Uncover system problems
Uncover experimental design problems
Uncover problems with materials
IMPORTANT IMPORTANT IMPORTANT
Participant Profile
What types people do you want to participate?
All the same?
Participant Profile
What types people do you want to participate?
All the same?
All different?
Participant Profile
What types people do you want to participate?
All the same?
All different?
People with extreme characteristics vs. “normal” people
Participant Profile
What types people do you want to participate?
All the same?
All different?
People with extreme characteristics vs. “normal” people
Consider
Age
Gender
Technology experience
….
How many participants?
This can be a difficult question
What claims you are making
What is feasible
Some will drop out!
Length
How long should the study be?
Another difficult question….
Novelty
What are you asking participant to do?
Long term use vs. feasibility
Data Collection
How will you understand if you answered your research question?
Quantitative Data (logs, timing, # errors, …) Qualitative Data (interviews, surveys, …)
TRIANGULATE between multiple sources
Logging
You must have a plan going in about how you will use the log data
Risk of forgetting to log something important
Logging too much can create an analysis nightmare
Make a list of questions you expect to answer with log data
How many times did they upload a photo?
How many days did they use your prototype?
Logging
Question How/Notes1. How often was SPARCS running, when did it stop? # of logfiles, LogfileStart(0)
LogfileClosed
2. Is automatic previewing setup? When Settings:IsPreviewing
3. Is automatic publishing setup? When Settings:IsAutoPublishing
4. How many people’s feeds were they subscribed to? SPARCS_Feed ADD as well as number of unique Create View
5. When did they add or subtract feeds? SPARCS_Feed, add or remove tags
6. What type of calendar feed are they using Calendar_Type, Settings:CalendarSource tag
Qualitative Data
Surveys
Pre-survey
Post condition
Post-survey
Experience Sampling Methodology
Small set of question
Event triggered, random….
Diaries
Interviews (semi-structured, structured)
Observation
Analyzing Qualitative Data
Affinity Diagramming
Coding of Comments
Inter-rater reliability
What if it doesn’t work?
There are many ways a study can fail
Technical problems
They don’t like it
Nobody but you cares about usability problems
Brace yourself for this
Figure out what has to work and skimp other places
Comparison between prototypes
Pilot studies
Outline
Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
Example
CareNet Consolvo, Roessler,
Selton Intel Research Seattle
The CareNet Display: Lessons Learned from an In
Home Evaluation of an Ambient Display
Sunny Consolvo, Peter Roessler, & Brett E. Shelton
Intel Research Seattle November ’04
November '04 The CareNet Display * Intel Research Seattle 34
Meet Rita…
Rita is one of the elders we worked with 83 years old Lives alone Conditions: Mild dementia & Type 2 Diabetes
(takes insulin)
She is struggling to maintain her independence, but needs help…
November '04
The CareNet Display * Intel Research Seattle 35
Rita’s Care Network
Drastic life changer:daughter Hannah
Significant contributor:son Simon
Significant contributor:son Zack
Neighbor
Daughter-in-law
Part-time professional caregiver
Son
Hannah’s boyfriend
Daughter-in-law
November '04
The CareNet Display * Intel Research Seattle 36
The focus of this work
Improve the quality of life for ALL care network members, including the elder
Help members coordinate care activities
Ensure elder gets care she needs
Give time back to overburdened members
November '04 The CareNet Display * Intel Research Seattle 37
The CareNet Display
Interactive digital picture frame always-on Internet access from
wireless GPRS card
Augments photo with updates meds, meals, outings, activities,
mood, falls, & shared calendar
Goal: help local care network members provide daily care
November '04
The CareNet Display * Intel Research Seattle 38
Participants & Methodology
13 participants from 4 care networks In home Wizard of Oz deployments
i.e., phone calls instead of sensors
Interviews & surveys No special instructions on:
how or when to use it, or where to place it
November '04
The CareNet Display * Intel Research Seattle 39
Some Results
Drastic Life Changers:
time back, less stress, “meaningful” conversations
Significant Contributors:
increased awareness of what others contribute
Peripherally Involved Members:
gave them something to talk about
Elders:
changed minds about how open they would be
November '04 The CareNet Display * Intel Research Seattle 40
Challenges: Sensor data with a “human touch” Participants are afraid sensor data will be too impersonal How can we provide this “human touch” without adding to the
responsibilities of already overburdened network members?
November '04
The CareNet Display * Intel Research Seattle 41
Early & in situ evaluations
Early evaluations help us learn where to concentrate our technology development
In situ evaluations provide us with valuable insight into how new tools are used and what effects they have on the intended users
November '04
The CareNet Display * Intel Research Seattle 42
For more information…
Publications: S. Consolvo, P. Roessler, & B.E. Shelton, "The
CareNet Display: Lessons Learned from an In Home Evaluation of an Ambient Display," Proceedings of the 6th Int'l Conference on Ubiquitous Computing: UbiComp '04 (Sep 2004), pp.1-17.
S. Consolvo, P. Roessler, B.E. Shelton, A. LaMarca, B. Schilit, & S. Bly, "Technology for Care Networks of Elders," IEEE Pervasive Computing Mobile and Ubiquitous Systems: Successful Aging, Vol. 3, No. 2, (Apr-Jun 2004), pp.22-29.
Lots of other examples…
SuperBreak Morris, Brush, Meyers
Microsoft Research
ButterflyNet Yeh, Liao, Klemmer, Guimbretière, Lee, Kakaradov, Stamberger,
Paepcke Stanford, Maryland
Wearable Jersey DisplayPage and Moere
University of Sydney
UbiFitConsolvo, McDonald, Toscos,
Chen, Froehlich, Harrison, Klasnja, LaMarca, LeGrand,
Libby, Smith, LandayIntel Research Seattle
Outline
Introduction
Types of Studies
Study Design
Example
Ten Mistakes to Avoid
Your Turn
#10 Not enough people involved
It takes a village
Huge time commitment, 24 hour support for field studies
Do
Include multiple people
#9 Not being prepared
Don’t want to realize at the end of the study that you forgot to do something important
Do
Study design document
Research Question
Participant Profile
Methodology (within/between etc)
Timeline
Data collection
Pilot studies
WARNING: If you do not plan, you plan to fail!
#8 Not enough time for logistics
Everything takes time…. (more than you think)
Recruiting
Installation (e.g. 16 people X 2 hours = )
Support
Do
Allow plenty of time for the study
Have enough people
#7 Seeing what you want to see
We all want our prototypes to be popular
Do:
Think carefully about how you discuss the technology with participants
Avoid leading questions
Stay close to the data and find multiple support for conclusions
Use neutral language, “Tell me more,” “ummm”
Don’t get defensive
#6 Being judgmental
Users are always “right”
They may say things that are offensive, objectionable, etc.
Do:
Leave your opinions at home
Collect feedback
Use neutral language
#5 Not monitoring usage
Don’t want to find out at the end of the study that people were not using the prototype
Do:
“phone home” messages
Server logs
Have a plan about when and how you might intervene
#4 Not collecting a variety of data
Hard to understand logs without interviews
Hard to know if what someone says in an interview matches their use without logs
Do
Interviews
Surveys
Logs
Support for your findings in multiple ways gives you (and reviewers) more confidence
#3 Prototype is not robust enough
Technical challenges are a distraction
You are making a “product”
Do
Usability studies
Pilots with friendly folks
#2 Making inappropriate claims
One of the biggest and most common mistakes
Your study is one data point
Do:
Being careful with your language
Include a “limitations” section
Tell a clear story (don’t have to tell every finding)
#1 Not having a clear research question
Hard to explain the choices you made
Hard to explain your findings
May end up focusing on usability problems
Do
Have a clear research question
Acknowledgments
Sunny ConsolvoIntel Research
Seattle
Beverly Harrison Intel Research
Seattle
Ed CutrellMSR India
User Studies Are Fun!
Always learn something Often surprising! Inspire you
Questions???
References
Brush, Ubiquitous Computing Field Studies in Ubiquitous Computing Fundamentals. A.J. Brush, [email protected]
Green & Salkind Using SPSS for the Windows and Macintosh: Analyzing and Understanding Data (nth Ed.)
APA Publication Manual http://www.apastyle.org/manual/index.aspx
HCI textbooks Preece, J., Rogers, Y., and Sharp, H. Interaction design: beyond
human-computer interaction. 2002. John Wiley & Sons, Inc
Top Related