“Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani...
48
“Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani Medhi 3 Edward Cutrell 3 William Thies 3 1 University of Washington 2 San Jose State University 3 Microsoft Research India
-
Upload
marion-grant -
Category
Documents
-
view
214 -
download
1
Transcript of “Yours is better!” Participant Response Bias in HCI Nicola Dell 1 Vidya Vaidyanathan 2 Indrani...
“Yours is better!”Participant Response Bias in HCI
Nicola Dell1
Vidya Vaidyanathan2
Indrani Medhi3
Edward Cutrell3
William Thies3
1University of Washington2San Jose State University3Microsoft Research India
HCI is expanding and diversifying
2
HCI is expanding and diversifying
There are large differences between the people being studied and those studying them 3
Demand characteristics are aspects of a study that convey the investigator’s
hypothesis to participants
4
Demand characteristics may result in participant response bias
5
Demand characteristics may result in participant response bias
6
Demand characteristics may result in participant response bias
Response bias can influence the outcome of studies with human participants
7
What about demand characteristics in HCI?
8
How do social and demographic factors influence demand
characteristics?
9
Let’s design an experiment to explore demand characteristics in HCI
10
Let’s design an experiment to explore demand characteristics in HCI
11
Completely identical video clips
Observe participants interacting with technological devices
12
Interview Script“Thank you for participating in my experiment. I am a computer scientist and I'm trying to improve video players on mobile phones. I want you to watch a short video on these two phones and tell me which one looks better, or if they look the same. The same video will play on both phones, but this phone uses my new player [indicate phone]. Please tell me your honest opinion and please concentrate because I will play each video only once. Do you have any questions? Ok, watch this one first. This one uses my new player [play video]. Now watch this one [play video]. Which one do you think looks better or do they look the same? Why? Thanks very much!”
13
Two different interviewers
Foreign interviewer and local interviewer
14
Participants: Auto rickshaw drivers
250 male participants aged 19 to 72 (M=37, SD=11.2)15
Conditions tested with the identical video clip
Rickshaw Drivers
Foreign Interviewer (+ translator) 50 participants
LocalInterviewer 50 participants
16
Participant Responses
17
0
10
20
30
40
50
35
7 8Num
ber
of p
eopl
e
Foreign Interviewer
Preferred interviewer’s videoPreferred other videoThought videos looked the same
Participant Responses
18
0
10
20
30
40
50
19
9
22
0
10
20
30
40
50
35
7 8Num
ber
of p
eopl
e
Foreign Interviewer Local Interviewer
Num
ber
of p
eopl
e
Preferred interviewer’s videoPreferred other videoThought videos looked the same
Demand characteristics can result in substantial participant response bias
19
Demand characteristics can result in substantial participant response bias
Social and demographic factors influence the amount of response bias
20
21
What happens if the two clips are different?
What happens if the two clips are different?
21
What happens if the two clips are different?
21
What happens if the two clips are different?
21Interviewer always associated to the low-quality clip
Conditions tested with the degraded video clip
Rickshaw Drivers
Without association 50 participants
Foreign Interviewer (+ translator) 50 participants
LocalInterviewer 50 participants
22
0
10
20
30
40
50
4
38
8
Preferred low-quality videoPreferred high-quality videoThought videos looked the same
Participant Responses
23
Num
ber
of p
eopl
e
Without association
Preferred low-quality videoPreferred high-quality videoThought videos looked the same
Participant Responses
23
Num
ber
of p
eopl
e
0
10
20
30
40
50
27
19
4
Foreign Interviewer
Num
ber
of p
eopl
e
0
10
20
30
40
50
4
38
8
Without association0
10
20
30
40
50
18
29
3
The characteristics of the interviewer can determine the outcome of the study
24
Explaining participant responses
25
Explaining participant responses
“The quality of the background color and figures is too light in that player while the quality of the color
and graphics is better in [your] one. Will [your] new player be introduced in the market?”
26
Explaining participant responses
“I prefer this one (associated) because the other one wasn't clear at all. This one was fully clear. I
can definitely show you the difference if we watch them together. I've given you my honest opinion, so please don't be cross with me if it wasn't the
right one.”
27
Detailed and insightful participant comments can’t be taken at face value
28
What about other participant groups?
200 male university students aged 19 to 41 (M=25, SD=3.8)
29
Responses with university students
30
• Identical videos: 2:1 response bias
Responses with university students
30
• Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference
Responses with university students
30
• Identical videos: 2:1 response bias • Degraded videos: order of magnitude difference
• “I feel that in the newer version which you have coded, whenever there was a significant color contrast between two parts of an image, your version was somewhat smoother and less pixelated.”
There is still substantial response bias in studies with peers
32
Limitations
33
Limitations
• Presence of a translator for some interviews
34
Limitations
• Presence of a translator for some interviews• The effects of social status and ethnicity will
vary from culture to culture
35
Limitations
• Presence of a translator for some interviews• The effects of social status and ethnicity will
vary from culture to culture • Many other aspects of demand characteristics
36
What can we do to minimize demand characteristics in HCI?
37
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies
38
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution
39
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution • Minimize the differences in social status
between investigators and participants
40
What can we do to minimize demand characteristics in HCI?
• Be aware that response bias affects all studies• Dissociate from a particular design or solution • Minimize the differences in social status
between investigators and participants• Use triangulation to validate data collected
41
Demand characteristics influence the outcome of studies with human participants
42
“Yours is better!”Participant Response Bias in HCI
Nicola Dell1
Vidya Vaidyanathan2
Indrani Medhi3
Edward Cutrell3
William Thies3
1University of Washington2San Jose State University3Microsoft Research India
[email protected] http://cs.washington.edu/homes/nixdell
