Intro to HCI Methods and Experimental Design - UMD
Transcript of Intro to HCI Methods and Experimental Design - UMD
![Page 1: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/1.jpg)
1
Intro to HCI Methods and Experimental Design
Michelle Mazurek
Some slides adapted from Lorrie Cranor, Blase Ur, Marshini Chetty, Elaine Shi, Christine Trask, and Yu-Xiang Wang, Howard Seltman
![Page 2: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/2.jpg)
2
Today
• Reminder: reading reports should include both (all) readings, not just one
• Project logistics
• Presentation assignments, HW2 soon
• Finish up privacy, then start experimental design
![Page 3: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/3.jpg)
3
Option 2b: Computer reads for you
• Platform for Privacy Preferences (P3P)
• W3C specification for XML privacy policies
– Proposed 1996
– Adopted 2002
• Optional P3P compact policy HTTP headers to accompany cookies
• Goal: Your agent enforces your preferences
![Page 4: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/4.jpg)
4
Criticisms of P3P
• Too complicated, hard to understand
• Lacks incentives for adoption
– Only major companies?
![Page 5: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/5.jpg)
5
PrivacyFinder: P3P search engine
• Checks each search result for computer-readable P3P privacy policy, evaluates against user’s preferences
• Composes search result page with privacy meter annotations and links to “Privacy Report”
• Allows people to comparison shop for privacy
• http://privacyfinder.org/
![Page 6: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/6.jpg)
6
![Page 7: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/7.jpg)
7
![Page 8: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/8.jpg)
8
![Page 9: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/9.jpg)
9
Impact on decisionmaking
• Online shopping study conducted at CMU lab
• Participants buy with their own credit cards– Bought batteries and a sex toy
• Pay them a fixed amount; keep the change
• Result: When information is accessible, many people will pay (a little) more for privacy
J. Tsai, S. Egelman, L. Cranor, and A. Acquisti. The Effect of Online Privacy Information on Purchasing Behavior: An Experimental Study. WEIS 2007. S. Egelman, J. Tsai, L. Cranor, and A. Acquisti. 2009. Timing is Everything? The Effects of Timing and Placement of Online Privacy Indicators. CHI2009.
![Page 10: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/10.jpg)
10
P3P in Internet Explorer
• Implemented in IE 6, 7, 8, 9, 10 …
• “Compact policy” (CP)
• If no CP, reject third-party cookies
• Reject unsatisfactory third-party cookies
![Page 11: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/11.jpg)
11
No P3P syntax checking in IE
• Accepts bogus tokens, nonsense policies
• Valid:
• Also accepted:
P. Leon, L. Cranor, A. McDonald, and R. McGuire. Token Attempt: The Misrepresentation of Website Privacy Policies through the Misuse of P3P Compact Policy Tokens. WPES 2010.
AMZN Facebook does not have a P3P policy. Learn why here: http://fb.me/p3p
CAO DSP COR CURa ADMa DEVa OUR IND PHY ONL UNI COM NAV INT DEM PRE
![Page 12: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/12.jpg)
12
Microsoft uses a “self-declaration” protocol (known as “P3P”) dating from 2002 …. It is well known – including by Microsoft – that it is impractical to comply with Microsoft’s request while providing modern web functionality.
![Page 13: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/13.jpg)
13
Can policy agents ever work?
• Simplify the practices enough?
• Require users to specify their preferences?
– Learn them via AI?
• Incentives for broad adoption?
![Page 14: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/14.jpg)
14
Requirements for meaningful control
• Individuals must:– Understand what options they have– Understand implications of their options– Have the means to exercise options
• Costs must be reasonable– Money, time, convenience, benefits
![Page 15: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/15.jpg)
15
Option 3: The power of math
• Can we provide strong guarantees that don’t rely on good behavior from the data collector?
• Sort of!
• Differential privacy, invented by Cynthia Dwork
![Page 16: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/16.jpg)
16
Privacy and Justin Bieber
• Suppose you are handed a survey:– Do you like listening to Justin Bieber?– How many Justin Bieber albums do you own?– What is your gender?– What is your age?
• After analysis, results will be released publicly
– Do you feel safe submitting a survey?– Should you?
![Page 17: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/17.jpg)
17
Brief notation
----
----
----
----
----
surveys
----
----
----
----
----
----
data set
Q?
privatized analysis
esults public population
of interest
Pop I ⊆ Pop
di
DI = {di | i ∈ I }
Q( DI ) = R Q is the privatized
query run on the data set, and R is the result released to the public
![Page 18: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/18.jpg)
18
What do we want? (Privacy)
• My answer has no impact on the released results
• Any attacker looking at published R can’t learn anything new about me personally (high probability)
• Q(D(I-me)) = Q( DI )
• Pr[secret(me) | R] = Pr[secret(me)]
![Page 19: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/19.jpg)
19
Why can’t we have it?
• If individual answers had no impact, the results would be useless
• Trends in R may be true of me too. (If I am 15, do I like Justin Bieber?)
• By induction, Q(D(I)) = Q( DØ )
• Pr(secret(me) | secret(Pop) > Pr(secret(me))
![Page 20: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/20.jpg)
20
Why can’t we have it?
If an attacker knows a function about me dependent on the general population:
• I’m 2x average age
• I’m the majority gender
Then the attacker knows things about me even if I don’t submit a survey!
• age(me) = 2*mean_age)
• gender(me) = mode_gender
• mean_age = 16
• mode_gender = F
• age(me) = 32 AND gender(me) = F
![Page 21: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/21.jpg)
21
What can we have instead?
• The chance that the released result will be R is nearly the same, regardless of whether I submit a survey
• There is no (well, *almost* no) additional harm from submitting the survey
![Page 22: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/22.jpg)
22
Differential privacy
• If A=1, there is 0 utility (individuals have no effect)
• If A >> 1, there is little privacy
• A should be chosen by collector to be close to 1
Pr[Q(DI) = R] Pr[Q(DI±i) = R] ≤ A, for all I,i,R
![Page 23: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/23.jpg)
23
What this means
• Probability of result is nearly the same, regardless of whether I submit a survey
• How can anyone guess which world is true?
world where I submit a survey
world where I don’t submit a survey
Result R Pr[R] = X Pr[R] = Y X ≈ Y
![Page 24: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/24.jpg)
24
Popular misconception
• The attacker can’t learn anything about me from the results (protection against all harms)
• NOPE: Background information still applies. Attackers can use aggregate results.
![Page 25: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/25.jpg)
25
How to do it (high-level)
• Output perturbation: Return query answer plus some noise
• Input perturbation: Add noise to survey data before storing
• Perturbation of intermediate results
• Sample and aggregate– Ask Q over smaller samples; aggregate results
![Page 26: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/26.jpg)
26
Challenges
• Utility / privacy tradeoffs– May require really large datasets – May result in ridiculous answers
• Privacy budget depletion– Each query reduces what else can be asked
• How can this fit in with personal privacy (as opposed to data protection?)
• What does a differential policy mean in practice?
![Page 27: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/27.jpg)
27
Requirements for meaningful control
• Individuals must:– Understand what options they have– Understand implications of their options– Have the means to exercise options
• Costs must be reasonable– Money, time, convenience, benefits
How can/should these approaches (laws, notice/choice, technical solutions)
be balanced?
![Page 28: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/28.jpg)
28
HCI AND EXPERIMENTAL DESIGN
![Page 29: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/29.jpg)
29
Human-Computer Interaction (HCI)
• You are not the user! You know too much.
• Think about the user throughout design
• Involve the user
![Page 30: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/30.jpg)
30
![Page 31: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/31.jpg)
31
What is usable?
• Intuitive / obvious
• Efficient
• Learnable
• Memorable
• Few errors
• Not annoying
• Status transparentImage from http://www.xkcd.com
![Page 32: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/32.jpg)
32
Difficulties
• Many systems and platforms
• Users are different from one another
• Required standards (or no standards)
• Documentation won’t necessarily be read
• Performance
• Legal / time pressures
• Social and external factors
![Page 33: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/33.jpg)
33
Thinking about research studies
• What are you hoping to learn?
• What are your hypotheses?
– Sometimes listed explicitly in a paper
• What are your metrics for success?– More secure, quicker to use, more fun, etc.
• What are you comparing to?
• What data might be helpful?
![Page 34: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/34.jpg)
34
Broad types of studies
• Field study
• Laboratory study
• Online study
• (Measurement study)
![Page 35: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/35.jpg)
35
Quantitative vs. Qualitative
• Quantitative: you have numbers (timing data, ratings of awesomeness)
• Qualitative: you have non-numerical data (thoughts, opinions, types of errors)
![Page 36: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/36.jpg)
36
Types of studies
• Find out what people want:– Contextual inquiry– Interviews– Focus groups– Surveys– Diary study (prompt people)
• Find out what/how people think:
– Interviews– Surveys
![Page 37: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/37.jpg)
37
Types of studies
• Controlled experiments to test causation:– e.g., A/B testing
• Role-playing• Experiments in the field• Full-factorial design, or not
![Page 38: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/38.jpg)
38
Types of studies
• Expert evaluation of usability:– Cognitive walkthrough– Heuristic evaluation
• Usability test:– Laboratory (“think aloud”)– Online study– Log analysis
![Page 39: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/39.jpg)
39
Usability of Fruit
• https://www.youtube.com/watch?v=3Qg80qTfzgU
![Page 40: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/40.jpg)
40
Data to collect during experiments
• Independent vs. dependent variables
• Performance (time, success rate, errors)
• Opinions and attitudes
• Audio recording, screen capture, video, mouse movements, keystrokes
• Formative (initial) vs. summative (validate)
![Page 41: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/41.jpg)
41
Even more data to collect
• Demographics– Age, gender, technical background, income,
education, occupation, location, disabilities, first language, privacy attitudes, etc.
• Open-ended questions
• Preferences and attitudesPlease respond to the following statements:*This user interface was difficult to understand1- Strongly disagree 2- Disagree 3- Neutral 4- Agree 5- Strongly agree*This tool was fun to use1- Strongly disagree 2- Disagree 3- Neutral 4- Agree 5- Strongly agree
![Page 42: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/42.jpg)
42
Study designs
• Between subjects– Each participant tests 1 version of the system– You compare these groups– Groups should be similar (verify!)
• Within subjects– Every participant tests everything– Very important to randomize order!– Fewer participants
![Page 43: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/43.jpg)
43
Logistics for a study
• How many participants?– Statistical power– Time, budget, participants’ time
• What kind of participants?– Skills, background, interests– Their motivations– Often not a “representative sample”
• What do you need to build, if anything?– Prototype fidelity
![Page 44: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/44.jpg)
44
Prototype fidelity High-fidelity, “Wizard of Oz,” low-fidelity
![Page 45: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/45.jpg)
45
Think-aloud example (in pairs)
• Download and install software that lets you encrypt your email
– Verify that it is installed
• Things you can ask:
– What are you thinking now?– What do you expect to happen if you do X?– How did you decide to do that?
![Page 46: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/46.jpg)
46
Paper prototype example (in groups) • Draw a paper prototype of a tool to encrypt
emails sent on Gmail
– First step: Identify two tasks that you want to make sure are usable
![Page 47: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/47.jpg)
47
Principles for research design
• “The goal of any research design is to arrive at clear answers to questions of interest while expending a minimum of resources.” – Ramsey and Shafer
• Goal: Identify sources of experimental variation and try to minimize/control them
![Page 48: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/48.jpg)
48
1. Internal validity
• Avoid confounds!– Avoid criticism about causality
• How?
– Randomize condition/treatment assignment– Change only one variable at a time per condition– Use blinding– Use a control group
![Page 49: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/49.jpg)
49
2. Construct validity
• What do our metrics measure?– Is it what we intended?– Discuss: How would you measure tech-savviness?
![Page 50: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/50.jpg)
50
3. External validity
• What population does your sample represent?– Race, gender, age, nationality, education, others
• What environment does your sample represent?
– Carefully controlled study vs. real world
• This is especially critical for security research
– Security as secondary task– What you think you should do vs. what you actually do
![Page 51: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/51.jpg)
51
Promote power
• Within-subjects promotes power– But has other drawbacks, esp. in security research
• Is what you are measuring strong enough?
– Do you have enough participants?– Potential tradeoff: Generalizability for power
• Covariates: Measure possible confounds, include in analysis
• Blocking: Group similar subjects, distribute across conditions
http
://4.
bp.b
logs
pot.c
om/-F
uha1
-JX
xto/
T_ss
NrN
OD
tI/A
AA
AA
AA
AA
o0/L
Xcl
0Pxz
g40/
s160
0/th
epow
er.jp
g
![Page 52: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/52.jpg)
52
Determine use cases and goals
• What are the concrete tasks users should be able to accomplish?
– Based on understanding of users!
• Set realistic metrics
![Page 53: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/53.jpg)
53
Paper prototyping
• Don’t overthink, just make it
• Draw a frame on a piece of paper
– Make all menus, etc.
• Redesign based on feedback
• “Think-aloud”
![Page 54: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/54.jpg)
54
Paper prototype example
![Page 55: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/55.jpg)
55
Paper prototype example
![Page 56: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/56.jpg)
56
Usability prototyping for websites Site Maps Storyboards
Schematics Mock-ups
![Page 57: Intro to HCI Methods and Experimental Design - UMD](https://reader033.fdocuments.us/reader033/viewer/2022050612/62744446168eaf410147dc23/html5/thumbnails/57.jpg)
57