Culture & Survey Measurement - UIC | CUPPA | Survey ... and survey measurement Oct 2005.pdfCulture &...

34
Culture & Survey Measurement Timothy Johnson Survey Research Laboratory University of Illinois at Chicago

Transcript of Culture & Survey Measurement - UIC | CUPPA | Survey ... and survey measurement Oct 2005.pdfCulture &...

Culture & Survey Measurement

Timothy JohnsonSurvey Research LaboratoryUniversity of Illinois at Chicago

What is culture?

It is “the collective programming of the mind which distinguishes the members of one group from another” (Hofstede, Culture’s Consequences, 1980, p. 21)

What is culture? cont’d

The “shared elements that provide the standards for perceiving, believing, evaluating, communicating, and acting among those who share a language, a historic period, and a geographic location”(Triandis, American Psychologist, 1996).

Some dimensions of cultureIndividualism/collectivismUncertainty avoidancePower distanceLong vs. short term time orientationCultural tightness & complexityOpenness-to-change vs. ConservatismSelf-enhancement vs. Self-transcendence

Some dimensions of culture cont’d

Achievement vs. ascriptionNeutral vs. emotional expression of feelingsDiffuse vs. specific involvement in affairs of othersVertical vs. horizontal relationships

1. Reliability

2. Validity

3. Equivalence

Elements of social measurement in cross-cultural research

Types of equivalence1. Calibration 17. Factor 33. Measurement 49. Stimulus2. Complete 18. Factorial unit 50. Structural3. Conceptual 19. Formal 34. Metaphorical 51. Substantive4. Construct 20. Full 35. Metric 52. Syntactic5. Construct 21. Functional 36. Motivational 53. Technical

operationalization 22. Grammatical- 37. Normative 54. Text6. Content syntactical 38. Operational 55. Theoretical7. Contextual 23. Indicator 39. Pseudo 56. Translation8. Credible 24. Idiomatic 40. Psychological 57. True-score 9. Criterion 25. Instrument 41. Psychometric 58. Verbal10. Cross-cultural 26. Item 42. Relational 59. Vignette11. Cross-national 27. Language 43. Relative equivalence12. Cultural 28. Lexical 44. Response 60. Vocabulary13. Definitional 29. Linguistic 45. Scalar equivalence14. Direct 30. Literal 46. Scale15. Exact 31. Meaning 47. Semantic16. Experiential 32. Measurement 48. Situational

Two general forms of equivalence

• Procedural EquivalenceEmphasis on equivalent methods

• Interpretive EquivalenceEmphasis on equivalent meaning

Cognitive survey response model

Question interpretation

Memory retrieval

Judgment formation

Response editing

Question interpretation/ comprehension

Emic (culture specific)

Etic (pancultural)

Category fallacy

Assuming a question or concept is universally understood when in fact understanding is culturally conditioned

“In this question, what does the word ‘stress’ mean to you?”

0

5

10

15

20

25

30

AfricanAmerican

MexicanAmerican

Puerto Rican White

Health ProblemsSocial Problems

Comprehension difficulties by race/ethnicity

12.0% 12.6%13.5%

14.5%

0

2

4

6

8

10

12

14

16

Per

cen

t

AfricanAmerican

MexicanAmerican

Puerto Rican White

Comprehension difficulties by reading difficulty & race/ethnicity

0

5

10

15

20

Less than 6th grade 6th-8th grade Grade 9+

Reading difficulty level (U.S. grade level)

Pro

port

ion

wit

h c

ompr

ehen

sion

di

ffic

ult

ies

White

African American

Mexican American

Puerto Rican

0

5

10

15

20

Least Somewhat Most

Abstraction level

Pro

port

ion

wit

h c

ompr

ehen

sion

dif

ficu

ltie

s

White

African American

Mexican American

Puerto Rican

Comprehension difficulties by abstraction level & race/ethnicity

Comprehension difficulties for vague quantifier format by race/ethnicity

11.0% 12.5% 12.6%

6.6%

0

5

10

15

20

25

Per

cent

African American MexicanAmerican

Puerto Rican White

Comprehension difficulties for numeric question format by race/ethnicity

18.9% 20.0%21.2%

15.6%

0

5

10

15

20

25

Per

cent

AfricanAmerican

MexicanAmerican

Puerto Rican White

Comprehension difficulties for yes/no format by race/ethnicity

7.2% 6.7%7.7%

4.3%

0

5

10

15

20

25

Per

cent

AfricanAmerican

MexicanAmerican

Puerto Rican White

Memory retrieval

Episodic vs. semantic search strategies

Memory cues

Accessibility

Anchoring

Response formatting

Response styles

Judgment formation

Measurement artifacts in survey research

Extreme Response Styles

Acquiescent Response Style

May be misinterpreted as substantive differences across groups

Extreme response style

The tendency to select endpoints of a response scale

Acquiescent response style

The tendency to agree with survey questions, regardless of question format (also known as “yea-saying.”)

1. Strongly disagree2. Somewhat disagree3. Neither disagree nor agree4. Somewhat agree5. Strongly agree

Cross-cultural findings: Extreme response styles (ERS)*

More ERS in countries high in uncertainty avoidanceLess ERS in countries high in individualism

*van de Vijver (2004). Symposium on Cross-Cultural Survey Research, SRL, Urbana, IL.

Cross-cultural findings: Acquiescence (ACQ)*

More ACQ in countries high in uncertainty avoidance & power distance

Less ACQ in countries high in individualism

*van de Vijver (2004). Symposium on Cross-Cultural Survey Research, SRL, Urbana, IL.

Self presentation

Social desirability

Interviewer effects

Response editing

% uncomfortable discussing alcohol use with interviewers from same/different cultural groups

0

5

10

15

20

25

30

35

40

AfricanAmerican

MexicanAmerican

PuertoRican

White

Same CultureDifferent Culture

A. Question Development Phase

B. Questionnaire Pretesting Phase

C. Data Collection Phase

D. Data Analysis Phase

Available methods for addressing cross-cultural equivalence

Question development stage1. Expert consultation/collaboration

2. Ethnographic & other qualitative approaches

3. “Good” question-wording practices

4. “Good” translation practices

5. Facet analysis

Questionnaire pretesting phase

1. Cognitive interviews/structured probes

2. Comparative response scale calibration

3. Comparative behavior coding

4. Compare alternative data collection modes

Data collection phase

1. Use multiple indicators

2. Use both emic & etic questions

3. Respondent/interviewer matching

Data analysis phase1. Item analysis

2. Item response theory

3. Anchoring vignettes

4. Confirmatory factor analysis

5. Multidimensional scaling

6. Applying statistical controls

7. Identity-equivalence method

Thank You.

[email protected]