Gathering Usability Data
-
Upload
preston-nunez -
Category
Documents
-
view
49 -
download
1
description
Transcript of Gathering Usability Data
Gathering Usability Data
Observing users & subjective data
Directing Sessions
Issues:Are you in same room or not?Single person session or pairs of peopleObjective data -- stay detached
Fall 2006 PSYCH / CS 6750 2
Collecting Data
Data gatheringNote-takingAudio and video tapeInstrumented user interfacePost-experiment questions and interviews
Fall 2006 PSYCH / CS 6750 3
Collecting Data Identifying errors can be difficultQualitative techniques
Think-aloud - can be very helpfulPost-hoc verbal protocol - review videoCritical incident logging - positive & negativeStructured interviews - good questions
“What did you like best/least?”“How would you change..?”
Fall 2006 PSYCH / CS 6750 4
Observing Users
Not as easy as you think
One of the best ways to gather feedback about your interface
Watch, listen and learn as a person interacts with your system
Fall 2006 PSYCH / CS 6750 5
Observation
Direct In same room Can be intrusive Users aware of your
presence Only see it one time May use 1-way mirror to
reduce intrusiveness
Indirect Video recording Reduces intrusiveness,
but doesn’t eliminate it Cameras focused on
screen, face & keyboard Gives archival record,
but can spend a lot of time reviewing it
Fall 2006 PSYCH / CS 6750 6
Location
Observations may beIn lab - Maybe a specially built usability lab
Easier to controlCan have user complete set of tasks
In fieldWatch their everyday actionsMore realisticHarder to control other factors
Fall 2006 PSYCH / CS 6750 7
Challenge
In simple observation, you observe actions but don’t know what’s going on in their head
Often utilize some form of verbal protocol where users describe their thoughts
Fall 2006 PSYCH / CS 6750 8
Verbal Protocol
One technique: Think-aloudUser describes verbally what s/he is thinking and
doingWhat they believe is happeningWhy they take an actionWhat they are trying to do
Fall 2006 PSYCH / CS 6750 9
Think Aloud
Very widely used, useful techniqueAllows you to understand user’s thought
processes better
Potential problems:Can be awkward for participantThinking aloud can modify way user performs task
Fall 2006 PSYCH / CS 6750 10
Teams
Another technique: Co-discovery learning (Constructive interation)Join pairs of participants to work togetherUse think aloudPerhaps have one person be semi-expert (coach)
and one be noviceMore natural (like conversation) so removes some
awkwardness of individual think aloud
Fall 2006 PSYCH / CS 6750 11
Alternative
What if thinking aloud during session will be too disruptive?
Can use post-event protocolUser performs session, then watches video
afterwards and describes what s/he was thinkingSometimes difficult to recallOpens up door of interpretation
Fall 2006 PSYCH / CS 6750 12
Historical Record
In observing users, how do you capture events in the session for later analysis??
Fall 2006 PSYCH / CS 6750 13
Capturing a Session
1. Paper & pencilCan be slowMay miss thingsIs definitely cheap and easy
Fall 2006 PSYCH / CS 6750 14
Time 10:00 10:03 10:08 10:22
Task 1 Task 2 Task 3 …
Se
Se
Capturing a Session
2. Recording (audio and/or video)Good for talk-aloudHard to tie to interfaceMultiple cameras probably neededGood, rich record of sessionCan be intrusiveCan be painful to transcribe and analyze
Fall 2006 PSYCH / CS 6750 15
Capturing a Session
3. Software loggingModify software to log user actionsCan give time-stamped key press or mouse eventTwo problems:
Too low-level, want higher level eventsMassive amount of data, need analysis tools
Fall 2006 PSYCH / CS 6750 16
IssuesWhat if user gets stuck on a task?You can ask
“What are you trying to do..?”“What made you think..?”“How would you like to perform..?”“What would make this easier to accomplish..?”Maybe offer hints
Can provide design ideas
Fall 2006 PSYCH / CS 6750 17
Subjective Data
Satisfaction is an important factor in performance over time
Learning what people prefer is valuable data to gather
Fall 2006 PSYCH / CS 6750 18
Methods
Ways of gathering subjective dataQuestionnairesInterviewsBooths (eg, trade show)Call-in product hot-lineField support workers
(Focus on first two)
Fall 2006 PSYCH / CS 6750 19
Questionnaires
Preparation is expensive, but administration is cheap
Oral vs. writtenOral advs: Can ask follow-up questionsOral disadvs: Costly, time-consuming
Forms can provide better quantitative data
Fall 2006 PSYCH / CS 6750 20
Questionnaires
IssuesOnly as good as questions you askEstablish purpose of questionnaireDon’t ask things that you will not useWho is your audience?How do you deliver and collect questionnaire?
Fall 2006 PSYCH / CS 6750 21
Questionnaire Topic
Can gather demographic data and data about the interface being studied
Demographic data:Age, genderTask expertiseMotivationFrequency of useEducation/literacy
Fall 2006 PSYCH / CS 6750 22
Interface Data
Can gather data aboutscreengraphic designterminologycapabilitieslearningoverall impression...
Fall 2006 PSYCH / CS 6750 23
Question Format
Closed formatAnswer restricted to a set of choicesTypically very quantifiableVariety of styles
Fall 2006 PSYCH / CS 6750 24
Closed Format
Likert ScaleTypical scale uses 5, 7 or 9 choicesAbove that is hard to discernDoing an odd number gives the neutral choice in
the middle
Fall 2006 PSYCH / CS 6750 25
Characters on screen
hard to read easy to read 1 2 3 4 5 6 7
Other Styles
Fall 2006 PSYCH / CS 6750 26
LaTeX
FrameMaker
WordPerfect
Word
Rank from1 - Very helpful2 - Ambivalent3 - Not helpful0 - Unused
___ Tutorial___ On-line help___ Documentation
Which word processingsystems do you use?
Closed Format
Advantages Clarify alternatives Easily quantifiable Eliminate useless answer
Disadvantages Must cover whole range All should be equally
likely Don’t get interesting,
“different” reactions
Fall 2006 PSYCH / CS 6750 27
Open Format
Asks for unprompted opinionsGood for general, subjective information, but
difficult to analyze rigorouslyMay help with design ideas
“Can you suggest improvements to this interface?”
Fall 2006 PSYCH / CS 6750 28
Questionnaire Issues
Question specificity“Do you have a computer?”
LanguageBeware of terminology, jargon
ClarityLeading questions
Can be phrased either positive or negative
Fall 2006 PSYCH / CS 6750 29
Questionnaire Issues
• Prestige bias – People answer a certain way because they want
you to think that way about them
• Embarrassing questions• Hypothetical questions• “Halo effect”
– When estimate of one feature affects estimate of another (eg, intelligence/looks)
Fall 2006 PSYCH / CS 6750 30
Deployment
StepsDiscuss questions among teamAdminister verbally/written to a few people
(pilot). Verbally query about thoughts on questions
Administer final test
Fall 2006 PSYCH / CS 6750 31
Interviews
Get user’s viewpoint directly, but certainly a subjective view
Advantages:Can vary level of detail as issue arisesGood for more exploratory type questions which
may lead to helpful, constructive suggestions
Fall 2006 PSYCH / CS 6750 32
Interviews
DisadvantagesSubjective viewInterviewer can bias the interviewUser may not appropriately characterize usageTime-consuming
Fall 2006 PSYCH / CS 6750 33
Interview Process
How toPlan a set of questions (provides for some
consistency)Don’t ask leading questions
“Did you think the use of an icon there was really good?”
Can be done in groupsGet consensus, get lively discussion going
Fall 2006 PSYCH / CS 6750 34
Data Analysis
Simple analysisDetermine the means (time, # of errors, etc.) and
compare with goal values (coming up…)
DetermineWhy did the problems occur?What were their causes?
Fall 2006 PSYCH / CS 6750 35
Experimental Results
How does one know if an experiment’s results mean anything or confirm any beliefs?
Example: 20 people participated, 11 preferred interface A, 9 preferred interface B
What do you conclude?
Fall 2006 PSYCH / CS 6750 36
Hypothesis Testing
In experiment, we set up a “null hypothesis” to checkBasically, it says that what occurred was simply
because of chance
For example, any participant has an equal chance of preferring interface A over interface B
Fall 2006 PSYCH / CS 6750 37
Hypothesis Testing
If probability result happened by chance is low, then your results are said to be “significant”
Statistical measures of significance levels0.05 often usedLess than 5% possibility it occurred by chance
Fall 2006 PSYCH / CS 6750 38
Presentation Techniques
Fall 2006 PSYCH / CS 6750 39
0 20
Mean
low highMiddle 50%
Time in secs.
Age
Upcoming
AudioWeb
Fall 2006 PSYCH / CS 6750 40
Using the Results
How do you use the results of your evaluation?
How can you make your design better with this knowledge?
Fall 2006 PSYCH / CS 6750 41