Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face...

18
Richard Johnston University of British Columbia (ex University of Pennsylvania & National Annenberg Election Study) Presented to ELECDEM Conference University of Exeter 19 November 2009 Challenges in Survey-Based Electoral Research

Transcript of Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face...

Page 1: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Richard Johnston University of British Columbia (ex University of Pennsylvania

& National Annenberg Election Study)

Presented to ELECDEM Conference University of Exeter 19 November 2009

Challenges in Survey-Based Electoral Research

Page 2: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Representation Challenges

 Persons

 Opinions

 Processes

… tradeoffs among these

Page 3: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Modes of survey data collection   Mail   Face-to-Face   Telephone

  Web Comparisons combine considerations that are:   intrinsic to mode as such

  very critical to representing opinion

  byproducts of how mode constrains how samples are drawn   very critical to representing persons   quite critical to representing processes

Page 4: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Sampling I: Unit Nonresponse

  Coverage

Mail F TF Phone > Web (what is the frame?)

  Contact

Mail (but within-HH issues) Web (but so what?) Phone > FTF

  Compliance

FTF > Phone ? Web > Mail

  Nonresponse and bias

FTF > all others

Phone & Web > Mail

Phone vs Web? -- debate over mix of equal-probability of selection and self-recruitment

Representing persons

Page 5: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Sampling II: Power per respondent -- Clustering

FTF is expensive

samples are clustered spatially to minimize cost per interview

Representing persons

Page 6: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Sampling II: Power per respondent -- Clustering

Effective N of observations =

where: J = number of clusters

n = observations per cluster

= intraclass correlation =

= between-cluster variance

= within-cluster variance

Jn1+ (n −1)ρ

ρ

τ 2

τ 2 +σ 2

τ 2

σ 2

Representing persons

Page 7: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Sampling II: Power per respondent -- Clustering

Effective N of observations =

where: J = number of clusters

n = observations per cluster

= intraclass correlation =

= between-cluster variance

= within-cluster variance

Jn1+ (n −1)ρ

ρ

τ 2

τ 2 +σ 2

τ 2

σ 2

Representing persons

Page 8: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Sampling II: Power per respondent -- Clustering

Effective N of observations =

where: J = number of clusters

n = observations per cluster

= intraclass correlation =

= between-cluster variance

= within-cluster variance

Jn1+ (n −1)ρ

ρ

τ 2

τ 2 +σ 2

τ 2

σ 2

Representing persons

Page 9: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode and Q&A Factors intrinsic to mode:   Presence or absence of an interviewer

 mail and web: no interviewer   FTF and phone: interviewer

  but cf CAPI – can make some response private   but cf automated phone surveys – ditto

  Visual vs oral/aural

Representing opinions

Page 10: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Interviewer effects Interviewers can:

  assist respondent   “model professionalism” but   force the pace artificially   bend interpretation of items   produce respondent/interviewer interactions – e.g. race,gender   inhibit socially undesirable response

 Web and mail facilitate socially disfavoured opinions e.g. sexuality, opinion on sexuality, opinion on race

Representing opinions

Page 11: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Visual vs Oral/Aural   minimum of interpretation vs premium on literacy

  web: forget about political knowledge indicators

  Rank order effects:   visual first item   oral/aural last item

  Optimizing vs satisficing – measurement error

  non-differentiation

  acquiescence

  non-response (not distributionally neutral, cf Berinsky)

web response seems to be more optimized

Page 12: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode in the CSES Module 2

MODE OF INTERVIEW | Freq. Percent Cum.

----------------------------------------+-----------------------------------

1. FACE-TO-FACE | 46,172 71.86 71.86

2. TELEPHONE | 5,658 8.81 80.66

3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72

4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53

5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00

----------------------------------------+-----------------------------------

Total | 64,256 100.00

(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total

----------------------+-------------------------------------------------------+----------

00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116

03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769

05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225

07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514

10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482

12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674

15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200

15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022

15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823

20302002. CZECH REPUB | 948 0 0 0 0 | 948

20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026

24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196

25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000

27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000

27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023

34402004. HONG KONG ( | 582 0 0 0 0 | 582

34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200

35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446

37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367

37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212

38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439

39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977

41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500

41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000

48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991

52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574

55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741

57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052

60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032

60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200

61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794

62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303

62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801

64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913

64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496

70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002

72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212

75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060

75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418

82602005. GREAT BRITA | 0 0 0 0 860 | 860

84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066

----------------------+-------------------------------------------------------+----------

Total | 46,172 5,658 4,533 3,092 4,801 | 64,256

Page 13: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode in the CSES Module 2

MODE OF INTERVIEW | Freq. Percent Cum.

----------------------------------------+-----------------------------------

1. FACE-TO-FACE | 46,172 71.86 71.86

2. TELEPHONE | 5,658 8.81 80.66

3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72

4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53

5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00

----------------------------------------+-----------------------------------

Total | 64,256 100.00

(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total

----------------------+-------------------------------------------------------+----------

00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116

03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769

05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225

07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514

10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482

12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674

15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200

15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022

15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823

20302002. CZECH REPUB | 948 0 0 0 0 | 948

20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026

24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196

25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000

27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000

27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023

34402004. HONG KONG ( | 582 0 0 0 0 | 582

34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200

35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446

37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367

37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212

38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439

39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977

41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500

41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000

48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991

52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574

55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741

57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052

60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032

60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200

61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794

62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303

62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801

64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913

64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496

70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002

72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212

75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060

75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418

82602005. GREAT BRITA | 0 0 0 0 860 | 860

84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066

----------------------+-------------------------------------------------------+----------

Total | 46,172 5,658 4,533 3,092 4,801 | 64,256

Page 14: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode in the CSES Module 2

MODE OF INTERVIEW | Freq. Percent Cum.

----------------------------------------+-----------------------------------

1. FACE-TO-FACE | 46,172 71.86 71.86

2. TELEPHONE | 5,658 8.81 80.66

3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72

4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53

5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00

----------------------------------------+-----------------------------------

Total | 64,256 100.00

(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total

----------------------+-------------------------------------------------------+----------

00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116

03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769

05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225

07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514

10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482

12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674

15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200

15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022

15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823

20302002. CZECH REPUB | 948 0 0 0 0 | 948

20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026

24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196

25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000

27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000

27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023

34402004. HONG KONG ( | 582 0 0 0 0 | 582

34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200

35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446

37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367

37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212

38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439

39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977

41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500

41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000

48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991

52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574

55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741

57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052

60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032

60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200

61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794

62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303

62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801

64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913

64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496

70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002

72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212

75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060

75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418

82602005. GREAT BRITA | 0 0 0 0 860 | 860

84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066

----------------------+-------------------------------------------------------+----------

Total | 46,172 5,658 4,533 3,092 4,801 | 64,256

Page 15: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Mode in the CSES Module 2

MODE OF INTERVIEW | Freq. Percent Cum.

----------------------------------------+-----------------------------------

1. FACE-TO-FACE | 46,172 71.86 71.86

2. TELEPHONE | 5,658 8.81 80.66

3. SELF-ADMINISTERED (MAIL; SELF-COMPLE | 4,533 7.05 87.72

4. COMBINATION OF TELEPHONE AND SELF-AD | 3,092 4.81 92.53

5. COMBINATION OF FACE-TO-FACE AND SELF | 4,801 7.47 100.00

----------------------------------------+-----------------------------------

Total | 64,256 100.00

(NUMERIC POLITY) | 1. FACE-T 2. TELEPH 3. SELF-A 4. COMBIN 5. COMBIN | Total

----------------------+-------------------------------------------------------+----------

00802005. ALBANIA (20 | 1,116 0 0 0 0 | 1,116

03602004. AUSTRALIA ( | 0 0 1,769 0 0 | 1,769

05602003. BELGIUM (20 | 2,225 0 0 0 0 | 2,225

07602002. BRAZIL (200 | 2,514 0 0 0 0 | 2,514

10002001. BULGARIA (2 | 1,482 0 0 0 0 | 1,482

12402004. CANADA (200 | 0 0 0 1,674 0 | 1,674

15202005. CHILE (2005 | 1,200 0 0 0 0 | 1,200

15802001. TAIWAN (200 | 2,022 0 0 0 0 | 2,022

15802004. TAIWAN (200 | 1,823 0 0 0 0 | 1,823

20302002. CZECH REPUB | 948 0 0 0 0 | 948

20802001. DENMARK (20 | 2,026 0 0 0 0 | 2,026

24602003. FINLAND (20 | 1,196 0 0 0 0 | 1,196

25002002. FRANCE (200 | 0 1,000 0 0 0 | 1,000

27612002. GERMANY (20 | 0 2,000 0 0 0 | 2,000

27622002. GERMANY (20 | 0 0 1,023 0 0 | 1,023

34402004. HONG KONG ( | 582 0 0 0 0 | 582

34802002. HUNGARY (20 | 1,200 0 0 0 0 | 1,200

35202003. ICELAND (20 | 0 1,446 0 0 0 | 1,446

37202002. IRELAND (20 | 0 0 0 0 2,367 | 2,367

37602003. ISRAEL (200 | 0 1,212 0 0 0 | 1,212

38002006. ITALY (2006 | 1,439 0 0 0 0 | 1,439

39202004. JAPAN (2004 | 1,977 0 0 0 0 | 1,977

41002004. KOREA (2004 | 1,500 0 0 0 0 | 1,500

41702005. KYRGYZSTAN | 2,000 0 0 0 0 | 2,000

48402003. MEXICO (200 | 1,991 0 0 0 0 | 1,991

52802002. NETHERLANDS | 0 0 0 0 1,574 | 1,574

55402002. NEW ZEALAND | 0 0 1,741 0 0 | 1,741

57802001. NORWAY (200 | 2,052 0 0 0 0 | 2,052

60402006. PERU (2006) | 2,032 0 0 0 0 | 2,032

60802004. PHILIPPINES | 1,200 0 0 0 0 | 1,200

61602001. POLAND (200 | 1,794 0 0 0 0 | 1,794

62002002. PORTUGAL (2 | 1,303 0 0 0 0 | 1,303

62002005. PORTUGAL (2 | 2,801 0 0 0 0 | 2,801

64202004. ROMANIA (20 | 1,913 0 0 0 0 | 1,913

64302004. RUSSIA (200 | 1,496 0 0 0 0 | 1,496

70502004. SLOVENIA (2 | 1,002 0 0 0 0 | 1,002

72402004. SPAIN (2004 | 1,212 0 0 0 0 | 1,212

75202002. SWEDEN (200 | 1,060 0 0 0 0 | 1,060

75602003. SWITZERLAND | 0 0 0 1,418 0 | 1,418

82602005. GREAT BRITA | 0 0 0 0 860 | 860

84002004. UNITED STAT | 1,066 0 0 0 0 | 1,066

----------------------+-------------------------------------------------------+----------

Total | 46,172 5,658 4,533 3,092 4,801 | 64,256

Page 16: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Satisfaction with democracy N = 48662 SD = 0.27

Mode (base is FTF) No controls + demographics + country fe

Phone 0.01 (0.004) -0.00 (0.004) 0.09 (0.016)

Self - mail 0.08 (0.006) 0.02 (0.006) 0.17 (0.009)

Self + phone 0.12 (0.005) 0.10 (0.005) 0.21 (0.010)

Self + FTF 0.12 (0.004) 0.11 (0.004) 0.15 (0.011)

Entries are coefficients from OLS. Standard errors in parentheses. Source: CSES

Page 17: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

Persuade others N = 52751 pr(1) = 0.21

Mode (base is FTF) No controls + demographics + country fe

Phone 1.75 (0.06) 1.62(0.06) 2.87 (0.47)

Self - mail 16.06 (0.78) 18.89 (0.95) 15.20 (1.64)

Self + phone 3.33 (0.13) 3.67 (0.15) 0.36 (0.05)

Self + FTF 0.71 (0.03) 0.79 (0.04) 0.28 (0.04)

Entries are odds ratios from logistic regression. Standard errors in parentheses. Source: CSES

Page 18: Challenges in Survey-Based Electoral Research · Modes of survey data collection Mail Face-to-Face Telephone Web Comparisons combine considerations that are: intrinsic to mode as

A Plea for Information   “Administrative” in general, including for nonresponding

units   facilitation of estimating unit selection effects

  Care in coding of mode  what are the critical features?

  intrinsic   sampling

  primary sampling unit  what is the true power of the sample?