Field studies
description
Transcript of Field studies
![Page 1: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/1.jpg)
FIELD STUDIES
![Page 2: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/2.jpg)
User studies Ubicomp: people use technology Must conduct user studies Also:
Focus groups Ethnographic studies Heuristic evaulations Etc.
![Page 3: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/3.jpg)
User studies Laboratory studies:
Controlled environment Field (in-situ) studies
Real world
![Page 4: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/4.jpg)
Field studies Appropriate for ubicomp:
Abundant data Observe unexpected challenges Understand impact on lives
Trade-off: Loss of control Significant time and effort
![Page 5: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/5.jpg)
Three common types Current behavior Proof of concept Experience with prototype
![Page 6: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/6.jpg)
How to think about user studies?
Formulate hypotheses
![Page 7: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/7.jpg)
Research steps1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and
hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems
![Page 8: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/8.jpg)
What is a hypothesis? Proposing an explanation Theory or hypothesis? “This is just a theory.” Some theories we live by (“just” not
justified): Newton’s theory of motion Einstein’s theory of relativity Evolutionary theory
![Page 9: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/9.jpg)
Hypothesis Must be tentative Must predict
![Page 10: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/10.jpg)
Hypothesis Some criteria of scientificity
Self-consistent Grounded (fits bulk of relevant knowledge) Accounts for empirical evidence Empirically testable by objective
procedures of science General in some respect and to some
extent
![Page 11: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/11.jpg)
On proposing hypotheses Anomalous phenomena:
Strange and unfamiliar (Bermuda triangle) Familiar yet not fully understood (cognitive
load) Is there already an explanation?
![Page 12: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/12.jpg)
Types of hypotheses Incremental Fundamental shift:
Ptolemy (c. 90 – c. 168): geocentric cosmology
Copernicus (1473 – 1543): heliocentric cosmology
![Page 13: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/13.jpg)
And then came… Kepler (1571 – 1630): elliptical orbits
![Page 14: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/14.jpg)
Fundamental shift example Ulcer:
Stress? Spicy food? Bacteria.
![Page 15: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/15.jpg)
Types of proposed explanations Causes Correlation Causal mechanisms Underlying processes Laws Functions
![Page 16: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/16.jpg)
Proposing causal explanations Studies show that using a cell phone
while driving increases the probability of getting into an accident. Why is that so? Pick up ringing phone Dial number See but don’t perceive
![Page 17: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/17.jpg)
Effects not always there Cell phone + driving:
Usually no accident Only one of the factors
![Page 18: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/18.jpg)
Remote and proximate causes Cell phone + driving:
Attention shift → missed signal → accident Remote cause → proximate cause → effect
![Page 19: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/19.jpg)
Correlation A and B are correlated if:
A → B B → A C → A and C → B A combination of (some of) the above Coincidence
Correlation vs. causal relation: Correlation doesn’t imply causal relation Cannot determine cause direction (A → B or B
→ A)
![Page 20: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/20.jpg)
Correlation Positive, negative None found ≠ none exists Causal link → correlation:
May provide initial evidence for causal link Less explanatory value than facts about
causal links
![Page 21: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/21.jpg)
Causal mechanisms Mechanisms connecting remote causes
and their effects. E.g.:
Damaged artery in heart → clotting Clotting → blocked artery Blocked artery → heart attack Aspirin inhibits clotting → lower risk of heart
attack
![Page 22: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/22.jpg)
Underlying processes Photoelectric effect
![Page 23: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/23.jpg)
Photoelectric effect Einstein: 1921 Nobel Prize in Physics
![Page 24: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/24.jpg)
Laws General regularities in nature Universal:
F = ma Non-universal:
Statistical laws
![Page 25: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/25.jpg)
Functions What is the purpose of the
phenomenon?FOR SALE A prime lot of serfs or SLAVES GYPSY (TZIGANY) Through an auction at noon at the St. Elias Monastery on 8 May 1852 consisting of 18 Men 10 Boys, 7 Women & 3 Girls in fine condition
![Page 26: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/26.jpg)
Functions William Harvey (1578 – 1657):
Heart pumps blood through circulatory system
No modern instruments! Experiments with a number
of animals: Various fish, Snail, Pigeon, etc.
![Page 27: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/27.jpg)
Multiple methods together
Function → → causal mechanism → → underlying processes
National Ignition Facility(Dennis O’Brien @ UNH):Ignition with lasers → → Laser, target chamber → → Physics of nuclear fusion
![Page 28: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/28.jpg)
Multiple methods together Law → underlying processes Isaac Newton (1643 – 1727),
second law of motion:F = ma → Graviton?
![Page 29: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/29.jpg)
Ockham’s razor Crop circles: pranksters or aliens?
![Page 30: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/30.jpg)
Ockham’s razor William of Ockham (c. 1288 – c. 1348)
http://en.wikipedia.org/wiki/File:William_of_Ockham.png
![Page 31: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/31.jpg)
Do I have a hypothesis? Yes. Do you realize you do?
![Page 32: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/32.jpg)
How to think about user studies?
Formulate hypotheses
![Page 33: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/33.jpg)
Three common types Current behavior Proof of concept Experience with prototype
![Page 34: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/34.jpg)
Research steps1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and
hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems
![Page 35: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/35.jpg)
Current behavior Insights and inspiration:
State problem(s), goal(s) Propose hypotheses
Relatively long
![Page 36: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/36.jpg)
Current behavior – example 1
AJ Brush and Kori Inkpen, “Yours, mine and ours?...” (pdf) (2005 movie inspiring title)
Home technology: users share, etc.
![Page 37: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/37.jpg)
Current behavior – example 2 Schwetak Patel et al. “Farther Than You
May Think…” (pdf) Hypothesis: Mobile phone a proxy to
user location.
![Page 38: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/38.jpg)
Three common types Current behavior Proof of concept Experience with prototype
![Page 39: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/39.jpg)
Research steps1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and
hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems
![Page 40: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/40.jpg)
Proof of concept Technological advance:
Produce results: prototype Explain results: prototype
Relatively short
![Page 41: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/41.jpg)
Proof of concept – example 1 J. Sherwani et al., “Speech vs. Touch-
tone: Telephone Interfaces for Information Access by Low Literate Users” (pdf) (video)
Hypothesis: Speech better telephony interface than touch-tone for low literate users.
![Page 42: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/42.jpg)
Proof of concept – example 2
John Krumm and Eric Horvitz, “Predestination:…” (pdf)
Hypothesis: Destinations from partial trajectories.
Train/test algorithm on GPS tracks from 169 people
Used pre-existing data: Krumm and Horvitz, “The Microsoft Multiperson
Location Survey” Collecting original data a significant contribution Leverage!
![Page 43: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/43.jpg)
Three common types Current behavior Proof of concept Experience with prototype
![Page 44: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/44.jpg)
Research steps1. State problem(s)2. State goal(s)3. Propose hypotheses4. Propose steps to test hypotheses5. Explain how problem(s), goal(s) and
hypotheses fit into existing knowledge6. Produce results of testing hypotheses7. Explain results8. Evaluate research9. State new problems
![Page 45: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/45.jpg)
Experience with prototype Users’ interaction with technology:
Produce results: prototype Explain results: prototype
Relatively long
![Page 46: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/46.jpg)
Prototype an example! Others don’t care about:
Raw usage information Usability problems Intricate implementation details Etc.
Generalize! Scientific and good technical work
![Page 47: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/47.jpg)
Experience – example 1 C. Neustaedter, et al., “A Digital Family
Calendar in the Home:…” (pdf) (video) Hypothesis: At-a-glance awareness,
remote access are significant benefits. 4 households, 4 weeks each (Best Student Paper, Graphics Interface
2007)
![Page 48: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/48.jpg)
Experience – example 2 Rafael Ballagas et al., “Gaming Tourism:
…” (pdf) (video) Hypothesis: Learning through a game. 18 participants: 2 alone + 8 pairs (8 x 2
= 16)
![Page 49: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/49.jpg)
Study design Who is the consumer?
Manager(s) Industry, academic lab
Professor(s) E.g. thesis committee
Researchers E.g. advisor’s collaborators
Reviewers For paper, proposal, thesis
Funding agency Report on progress, proposal for funding
Public Friends, family, alumni, potential students, donors,
potential employers
![Page 50: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/50.jpg)
Study design How can I explain this to a layperson?
What is key? What can be omitted? How will I write this up?
Paper Thesis Report Blog post
Start writing paper/thesis/report/blog post at the beginning of the study.
![Page 51: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/51.jpg)
Study design Test hypothesis/hypotheses
![Page 52: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/52.jpg)
Testing hypotheses via user studies
User studies: Laboratory studies
Good: Control, easier to evaluate results Bad: Constraints
Field studies Good: Fewer constraints Bad: Less control, more difficult to evaluate
results
![Page 53: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/53.jpg)
Criteria Falsifiability:
Prediction fails = explanation isn’t correct Account for other factors!
Note: Criterion - singular Criteria - plural
![Page 54: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/54.jpg)
Criteria Verifiability:
Prediction successful = explanation is correct
Account for other factors!
![Page 55: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/55.jpg)
The meat of it… Battleship Potemkin
, 1925 film Rotten meat scene
![Page 56: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/56.jpg)
Why larvae in meat? Francesco Redi
(1626-1697) Generation of
insects, 1668 Causal
explanation: fly droppings
![Page 57: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/57.jpg)
Redi’s research Hypothesis:
Worms derived from fly droppings Testing hypothesis:
Two sets of flasks with meat: sealed and open
Prediction: worms only in open flask
![Page 58: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/58.jpg)
Falsifiability criterion Can anything cause a failed prediction
even if explanation is correct? Did the apparatus operate properly?
Tight seal? Meat not initially spoiled? Other?
![Page 59: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/59.jpg)
Verifiability criterion Can anything result in successful
prediction even if explanation is wrong? What if “active principle” in the air is
responsible for spontaneous generation? Modify experiment:
Replace seal with veil: Flies cannot reach meat Air in contact with meat
Modification helps meet verifiability criterion
![Page 60: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/60.jpg)
Verifiability criterion Experimental vs. control group:
Only difference in level of one independent variable
Redi’s experiment: Control: Open flasks Experimental: Veil-covered flasks
![Page 61: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/61.jpg)
Control: laboratory experiment Meat in veil-covered flasks? Creating control/experimental groups
often impossible without careful design/control
![Page 62: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/62.jpg)
Study design Test hypothesis/hypotheses Formulate in terms of:
Independent variables (multiple conditions) Dependent variables
Design: Within-subjects Between-subjects Mixed design
![Page 63: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/63.jpg)
Within-subjects design: example
Police radio UI: hardware Speech
Blog post, video
![Page 64: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/64.jpg)
Within-subjects design: example
Results in graphical form:
![Page 65: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/65.jpg)
Within-subjects design: example
Results in graphical form:
![Page 66: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/66.jpg)
Example: between-subjects design
Classical example: testing a drug
![Page 67: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/67.jpg)
Mixed design: example 1 SUI characteristics study Secondary task: speech control of radio 2 x 2 x 2 design:
SR accuracy: high/low PTT button: yes/no – ambient recognition Dialog repair strategy: mis-/non-
understanding
![Page 68: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/68.jpg)
![Page 69: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/69.jpg)
Mixed design: example 2 Motivation: PTT vs. driving performance Secondary task: speech control of radio 2 x 3 x 3 design:
SR accuracy: high/low PTT activation:
push-hold-release/push-release/no push PTT button: ambient/fixed/glove
Push-hold-release Push-release No-push
Ambient Fixed Glove Ambient Fixed Glove Ambient Fixed Glove
High
Low
![Page 70: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/70.jpg)
Control condition Baseline: e.g. no technology vs. later
introduced technology
![Page 71: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/71.jpg)
Considerations What will subjects do?
Normal behavior – may take long Scenarios
Augment existing or brand new? Augment: taking advantage of familiarity New: more control (fewer inherited
constraints) Simulate or implement?
E.g. WoZ
![Page 72: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/72.jpg)
Data to collect Qualitative
Insight into what participants did. How do participants compare? Did they do
what they thought they did? Use quantitative data.
Quantitative How did people behave? But why? Use qualitative data.
![Page 73: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/73.jpg)
Data to collect At least three types of data:
Demographic Usage Reactions
![Page 74: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/74.jpg)
Data to collect Run pilot experiments!
![Page 75: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/75.jpg)
Collecting data Logging Surveys Experience sampling Diaries Interviews Unstructured observation – ethnography
![Page 76: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/76.jpg)
Logging Plan ahead, not after the fact!
Testing hypotheses Don’t leave important data out Don’t save data you don’t need
Leverage logging: Everything OK?
E.g. Mike Farrar’s MS research: files appearing on server indicates apps OK
Explicit communication with server: “I’m OK!”
![Page 77: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/77.jpg)
Surveys Open-ended Multiple-choice Likert-scale
![Page 78: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/78.jpg)
Surveys Questions should allow positive and
negative feedback. Text clear to others?
Check! One question at a time!
“Fun and easy to use?” Length?
Don’t bore subjects to death. Standard questions (e.g. QUIS)?
Previously used questions?
![Page 79: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/79.jpg)
Example: Mike Farrar’s study Hypotheses:
Initialize grammar (video): From previous tags From tags by users with similar interests
Voice commands convenient way to tag photos (video)
Keyboard users will use voice less Low task completion: give up on voice
![Page 80: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/80.jpg)
Experience sampling (ESM) Short questionnaire Timing:
Random Scheduled Event-based
![Page 81: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/81.jpg)
Experience sampling (ESM) How often? How many? Relate to quantitative data?
![Page 82: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/82.jpg)
Diaries Similar to ESM
![Page 83: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/83.jpg)
Interviews Semi-structured:
List of specific questions + follow-up questions
Bring data E.g. Nancy A. Van House: “
Flickr and Public Image Sharing:…” Interviews + photo elicitation
![Page 84: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/84.jpg)
Interviews Neutral questions Negative feedback is OK (this is hard):
Don’t argue!
![Page 85: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/85.jpg)
Participants Follow IRB rules
![Page 86: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/86.jpg)
Participants Who to recruit?
Representative of intended users Not your friends, family, colleagues – bias! May need different types
Recruit sufficient numbers of each type
![Page 87: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/87.jpg)
Participant profile Age
E.g. age significant for driving Gender Technology use and experience Other
Eye tracker studies: no glasses
![Page 88: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/88.jpg)
Number of participants Between-subjects usually requires more
than within-subjects Proof-of-concept: typically fewer and
many types Longer study: may be able to use fewer Time commitment per participant is
significant! Recruit (Craigslist), organize, train, run,
transfer data, process data Participants will drop out – recruit extra
Counterbalancing may not work out
![Page 89: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/89.jpg)
Compensation Don’t try to save on this! Driving simulator lab study cost example
1 graduate student year at UNH ≈ $50k Software maintenance fees per year ≈
$20k Trip to conference ≈ $2k PC or laptop ≈ $2k $20 x 24 participants ≈ $0.5k (less than
1%)
![Page 90: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/90.jpg)
Compensation Must not affect data
E.g. in image tagging study if we paid per picture: More data Unrealistic as interactions are for money not for
value of prototype
![Page 91: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/91.jpg)
Compensation Leverage if you can:
Latest driving simulator lab study in collaboration with Microsoft Research: Use Microsoft software as compensation
![Page 92: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/92.jpg)
Data analysis Test hypotheses Use multiple data types Tell a story
![Page 93: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/93.jpg)
Data analysis Statistics:
Descriptive Inferential
![Page 94: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/94.jpg)
Descriptive statistics Level of measurement:
Nominal Ordinal Interval
![Page 95: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/95.jpg)
Descriptive statistics Level of measurement:
Nominal Ordinal Interval
![Page 96: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/96.jpg)
Level of measurement Nominal:
Unordered categories E.g. yes/no Valid to report :
Frequency
![Page 97: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/97.jpg)
Level of measurement Ordinal:
Rank order preference without numeric difference
E.g. responses on Likert scale Five of the eight participants strongly agreed or
agreed with the following statement: “I prefer to have a GPS screen for navigation.”
Valid to report : Frequency Median Some people report means but what is the mean
of “strongly agree” and “strongly disagree”?
![Page 98: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/98.jpg)
Level of measurement Interval:
Numerical differences significant E.g. age, number of times an action
occurred, etc. Valid to report:
Sum Mean Median Standard deviation (outliers?)
![Page 99: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/99.jpg)
Outliers in interval data
![Page 100: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/100.jpg)
Inferential statistics Significance tests
t-test ANOVA Many others
Which to use: depends on data
![Page 101: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/101.jpg)
Significance test: example 1 To assess the effect of different
navigation aids on visual attention, we performed a one-way ANOVA using PTD as the dependent variable. As expected, the time spent looking at the outside world was significantly higher when using spoken directions as compared to the standard PND directions, p<.01. Specifically, for spoken directions only, the average PDT was 96.9%, while it was 90.4% for the standard PND.
![Page 102: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/102.jpg)
Significance test: example 2
-5
0
5
10
15
20
60-80 80-100 100-120 120-140 140-160
PDT
on st
anda
rd P
ND
[%]
distance from previous intersection [m]
… PDT on the PND screen changes with the distance from the previous intersection… significant main effect, p<.01…
![Page 103: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/103.jpg)
Significance test: example 3 Randomization test
Kun et al. (pdf) Idea from Veit et al. (pdf)
![Page 104: Field studies](https://reader035.fdocuments.us/reader035/viewer/2022070500/56816906550346895de01881/html5/thumbnails/104.jpg)
Significance test: example 3
0
5
10
15
20
25
30
35
0 1 2 3 4 5 6 7 8
Rstw
[deg
rees
^2 ]
lag [seconds]
standard
p = 0.05
spoken only