RTI International

RTI International is a trade name of Research Triangle Institute. www.rti.org

Cognitive Interviewing in Online Modes: a Comparison

of Data Collected in Second Life & Skype

Jodi E. Swicegood

Elizabeth F. Dean

Brian F. Head

Michael D. Keating

RTI International

1

RTI International

Study Goals

Pilot test of cognitive interviewing in Skype and Second

Life (SL)

Cognitive testing of questionnaire about SL avatars &

identity

Recruitment – Evaluate recruitment potential of Facebook and Craigslist at recruiting

respondents to virtual cognitive interviews

– Assess feasibility of recruiting users of the virtual world SL

2

RTI International

Cognitive Interviewing

A pretesting technique used to identify questionnaire

problems, such as:

– Comprehension

– Vague or inconsistent instructions

– Logic or flow

– Participant burden

Often conducted in a lab setting, sometimes on-site

Challenges:

– Recruitment

– Identifying appropriate sample

– Selecting appropriate mode

3

RTI International

Virtual Cognitive Interviewing with New Technologies

Shared online spaces introduce a range of options for

cognitive interviewing methodology.

– Skype video calling enables face to face, private cognitive

interviews in any location

– Virtual worlds like Second Life allow users to conduct and

participate in cognitive interviews through avatars

Pros

– Access to wider population without travel

– Convenience for researcher & participant (e.g. scheduling)

– Reduced time and costs

Considerations

– Technical requirements; technical knowledge

4

RTI International

Research Questions

1). How do these virtual interviewing modes compare in

terms of...?

Number

Type

Severity of errors – total # of problems observed across

measures

2). Does data quality differ across modes?

5

RTI International

Data Collection Process

Collected data for 2 ½ months

Conducted 40 interviews w/ SL users

– 21 in SL

– 16 Skype

– 3 In-Person

All participants were contacted in SL 6

RTI International

Second Life Interviews

7

RTI International

Skype Interviews

8

RTI International

Measures of Data Quality

Audio Quality (1=low; 2=medium; 3=high)

Video Quality (1=low; 2=medium; 3=high)

Participant Disengagement (1=disengagement)

Nonverbal Cues (Count of the # of nonverbal cues)

Types of Problems (1=problem identified)

– Comprehension

– Retrieval

– Judgment

– Response

Total Number of Problems (Sum)

9

RTI International

Participant Disengagement

Mode

Total

Observations of

Participant

Disengagement

Mean Number of

Participant Disengagement

(SD)

Second Life (n = 17) 14 0.82 (1.67)

Skype (n = 16) 18 1.13 (1.75)

In Person (n = 2) 0 0 (0)

Total Interviews

(n = 35)

32 0.91 (1.65)

10

RTI International

Nonverbal Cues

Mode

Total Observations

of Nonverbal Cues

Mean Number

Nonverbal Cues (SD)

Second Life (n = 17) 9 0.53 (0.80)

Skype (n = 16) 34 2.13 (3.50)

In Person (n = 2) 19 9.50 (13.44)

Total Interviews

(n = 35)

62 1.77 (3.92)

11

RTI International

Problems in the 4 Stages of the Response Process

30

8 5

42 43

4 8

66

13

1 3 8

0

10

20

30

40

50

60

70

# O

F O

BS

ER

VA

TIO

NS

MEASURE

Question Problems per Mode

SL (n=17)

Skype (n=16)

In Person (n=2)

12

RTI International

Discussion

Relationship between presence and number of problems

observed

– In person (most presence): total problems: 29; mean: 14.5

– Skype : total problems: 114; mean 7.13

– SL (least presence): total problems: 94; mean: 5.53

We believe cognitive interviews are feasible

– Skype: used with global or otherwise widely dispersed

populations; testing questions for an online panel

– SL: testing survey that contains sensitive topics (i.e. online role

playing & gaming; online relationships).

13

RTI International

Conclusion

– Use of virtual cognitive interviewing should be

weighed according to total costs & potential benefits

– Rapid growth in web camera & software technology

may decrease the current gap in presence between

face-to-face & virtual modes

– Future research should continue testing these

platforms (and the ones that will be created in time for

our next study) adding greater controls (i.e. random

mode assignments; larger sample sizes)

14

RTI International

Questions?

15

RTI International

Contact Me

Jodi Swicegood

Research Methodologist

swicegood@rti.org

SL: Katherine Cunningham

Skype: jodiswicegood

SurveyPost blogs.rti.org/surveypost

@SurveyPost