User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM...
Transcript of User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM...
![Page 1: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/1.jpg)
User Testing & Experiments
![Page 2: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/2.jpg)
Objectives
• Explain the process of running a user testing or experiment session.
• Describe evaluation scripts and pilot tests and explain why they are important
• Explain the difference between user testing and experiments.
• Describe the major building blocks of an HCI experiment.
• Identify flaws in experimental designs and suggest better alternatives
![Page 3: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/3.jpg)
Procedure for User Testing
From http://www.htc.net/~joegrant/web_pres_trial_v2/sld001.htm
![Page 4: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/4.jpg)
![Page 5: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/5.jpg)
![Page 6: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/6.jpg)
![Page 7: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/7.jpg)
![Page 8: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/8.jpg)
![Page 9: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/9.jpg)
![Page 10: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/10.jpg)
![Page 11: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/11.jpg)
Welcoming the user: why & how?
• Why? If they are at ease they will be – More comfortable sharing honest opinions – Less stressed during the tasks
• How? – Thank them for taking the time to help you – Explain that you are testing the interface,
not them
![Page 12: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/12.jpg)
Controlling bias
Now you get to work with pop-up menus. I think you will really like them... I designed them myself!
How else might the experimenter bias a user test?
![Page 13: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/13.jpg)
Evaluation Script
• A script that you read instructions from – Ensures all participants receive exactly the
same instructions – Ensures you don’t forget anything
• Can be awkward to read instructions – But you can explain to participants why you
are doing this
![Page 14: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/14.jpg)
Pilot test
• A “practice run” of your usability test or experimental session – Gives you practice – Evaluates procedures, tasks, instructions,
scripts, questionnaires, timing etc. – Gives you initial data – is your data
answering the questions you care about? • Participants don’t need to exactly match
target users
![Page 15: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/15.jpg)
Experiments vs. User testing
• Experiments test hypotheses -- investigate the relationship between two or more variables.
• User testing checks that a system is usable by the intended user population for their tasks.
![Page 16: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/16.jpg)
User testing • Aim: improve
products • Few participants • Results inform design • Not perfectly
replicable • Controlled conditions • Results reported to
developers
Research experiment • Aim: discover
knowledge • Many participants • Results validated
statistically • Replicable • Strongly controlled
conditions • Scientific paper reports
results to community
![Page 17: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/17.jpg)
When to Do an Experiment? • Comparison • Important consequences of
decision • Prediction uncertain • For research
– Important to show causality
![Page 18: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/18.jpg)
Exercise 1 Experiment or user testing?
– Will comparison of images be less accurate if done successively on a single screen as compared to simultaneously on two screens?
– Can radiologists learn to use a trackball and do they find it to be a satisfactory input device?
– Do radiologists prefer linearly or hierarchically organized lists of x-rays?
– Are the number of consultations between radiology and emergency increased by increased availability of workstations in both sites?
![Page 19: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/19.jpg)
Experiment Terminology Hypothesis: A tentative prediction that is testable
Independent variable: systematically manipulated by the experimenter Dependent variable: what you measure
Confounding factor: varies with the independent variable, making it impossible to attribute causality to the independent variable
![Page 20: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/20.jpg)
Exercise 2: Hypothesis
Improve this hypothesis:
• Popup menus are fast to use
File Edit View Insert
New Open
Close Save
File Edit
View Insert
New Open
Close Save
![Page 21: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/21.jpg)
User performance is faster when selecting a single item from a pop-up vs. a pull down menu of 4 items, regardless of the subject’s previous expertise in using a mouse or using the different menu types
![Page 22: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/22.jpg)
Exercise 3: Variable Types Name the independent & dependent variables:
– Is reading speed faster from a high resolution or low resolution screen?
– Will radiologists prefer a mouse, lightpen, or trackball as a pointing device?
– Can air traffic controllers enter data faster with a QWERTY or ABC keyboard?
![Page 23: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/23.jpg)
Exercise 4: Confounds Name the intended independent variable and the confounding variable. Suggest a better design.
– In a study to compare legibility of 2 visual displays, one display is located near a window and the other is in a windowless room.
– To determine whether computer training leads to more positive opinions of computers, attitudes of students in a 1st year computer course are compared to attitudes of 1st year students who are not taking a computing course.
![Page 24: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/24.jpg)
Exercise 4 continued
Volunteers compared 2 command selection techniques: function keys and mouse. All volunteers used the function key approach followed by the mouse approach.
An experiment compares three colour scales: a blue-yellow scale, a greyscale, and a rainbow scale. All participants complete the task first with the blue-yellow scale, then with the greyscale, and then with the rainbow scale.
![Page 25: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/25.jpg)
Experimental Designs Between-subjects: each participant sees only one condition
Within-subjects: each participant sees all conditions
*** Need to counter-balance order of conditions
Matched: Between-subjects design where the groups are matched on some factor (e.g. IQ or computer experience)
Mixed: Some factors between, some within
![Page 26: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/26.jpg)
Advantages Disadvantages Between-Subjects No order effects Need more participants
Individual differences Within-Subjects Fewer participants
No group differences Order effects
Matched Individual differences less than Between-Subjects
Matching may be imperfect May not know best matching factor
![Page 27: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/27.jpg)
Exercise 4 continued Do people remember information better with 2D displays or 3D displays? Participants were randomly assigned to 2D and 3D display groups. Instructions & experimental task were the same for both groups.
![Page 28: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/28.jpg)
Exercise 4 continued
An experiment compares (1) maps with an overview window to (2) maps without an overview. So that they do not repeat the exact same task twice, participants use a map of California with (1) and a map of Oregon with (2).
![Page 29: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/29.jpg)
Ways to control confounding factors?
• Randomize • Counter-balance • Hold constant • Eliminate • Match or balance
![Page 30: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/30.jpg)
Exercise 5: Choose a Design
Design an experiment to test the effects of the following factors on the speed of selecting an item in a menu:
– Menu type: pop-up or pull-down – Menu length: 3, 6, 9, 12, 15 items – Subject type (expert or novice)
(Decide whether each factor will be within or between-subjects & how to control order effects.)
![Page 31: User Testing & Experiments · 15_eval4.ppt Author: Melanie Tory Created Date: 3/16/2011 4:34:42 PM ...](https://reader034.fdocuments.us/reader034/viewer/2022042914/5f4e193c51abe6409d6f11fb/html5/thumbnails/31.jpg)
Key Points
• Run pilot tests • Set participants at ease • Use experiments when you need to
show causality. Otherwise user testing may suffice.
• Experiments must be defined very carefully to avoid confounds.