The development of push-to-web surveys in the UK · –2015 Japanese Census –2016 Canadian and...

© 2016 Ipsos. All rights reserved. Contains Ipsos' Confidential and Proprietary information and may

not be disclosed or reproduced without the prior written consent of Ipsos.

1Document Name Here | Month 2016 | Version 1 | Public | Internal Use Only | Confidential | Strictly Confidential (DELETE CLASSIFICATION)

Gerry Nicolaas & Patten Smith

NatCen-ESS ERIC-City Methodology Seminar Series

25 October 2017

London, UK

push-to-web surveys in the UK

The development of


Structure of presentation

1. Background: emergence of push-to-web methods

2. Key methodological challenges

3. The future


Focus of presentation

Focus on surveys with four main characteristics:

1. Random Probability sampling

2. General Population coverage

3. Online data collection methods

4. Offline contact / recruitment methods


4

Paste co-

brand

logo here

Background


Pressure to move high quality surveys online

Push and pull factors

Key push factors:

– Face-to-face survey declining response rates

– Face-to-face increasing survey costs / interviewer recruitment difficulties

– Concerns about survey budgets

– Reduced viability of RDD alternatives – response rates and mobile only

– Pressures from methodologically uninformed budget holders

– Govt. digital transformation (work stream 1: moving survey data collection online, changing

existing processes so that survey data is predominantly collected using online methods rather

than existing use of paper, telephone and face-to-face interviews; non-online methods would

only be used where there is an exceptional reason to do so)


Pressure to move high quality surveys online

Key pull factors:

– Increasing online coverage – now at 90%

– Increasing public expectation of online contact

– Low cost relative to face-to-face interviewing

– Methodological evidence that low response rates can be less damaging than feared


Why push-to-web?

Good random probability surveys require high coverage sample

frames

In UK, USA and other countries, good frames lack email addresses,

with consequence that…

…we have to make contact via postal address and persuade

(‘push’) respondents to respond online

Thus was born the push-to-web survey defined as use of offline

contact modes to encourage people to go online and complete

a web questionnaire


Push-to-web surveys

Several terms in use: commonly ‘web-push’; Kantar ‘ABOS’

Offline recruitment critical to definition. Recruitment methods include:

1. later waves of an offline panel (USoc)

2. piggy-backing on an existing offline survey (NatCen panel, CRONOS)

3. contacting a fresh sample by post, f2f or telephone

Main focus here on 3 - but overlap in issues.

Usually also provide alternative modes of data collection to web

non-respondents to improve response rate and coverage


Emergence of push-to-web surveys outside UK

Push-to-web surveys burgeoning worldwide, for example:

– American Community Survey

– 2015 Japanese Census

– 2016 Canadian and Australian Censuses

Methodological leadership emerged from Dillman and

colleagues’ research programme

Dillman, D., A. (2017). The promise and challenge of pushing respondents to the Web in

mixed-mode surveys. Survey Methodology, 43, 1, 3-30


Emergence of push-to-web surveys in the UK

Developing fast in UK. Examples include:

– ESS web experiment in 2012 (with f2f follow-up)

– Community Life web experiments 2012-15 (now replacing f2f)

– Active Lives Survey since 2015 (replacing RDD)

– Plus various one-off studies

ONS Social Survey Transformation

– Development & testing of online data collection for the LFS

– Development & testing of online data collection for the 2021 Census


11

Paste co-

brand

logo here

Key methodological challenges

1. Respondent selection

2. From postal contact to online completion

3. Overall response rates

4. Measurement equivalence across modes and devices


12

Paste co-

brand

logo here

1. How to select

respondents within

households?


Risk of self-selection bias

1. Multi-household addresses

– About 1% of PAF addresses contain more than one household

2. Respondent selection within households

– 65% of households include more than one adult

– No information available on the number and age of household

members

when using a sample of addresses and postal contact


Respondent selection approaches with postal contact

(1) Last/Next Birthday methods

2012 ESS Mixed Mode Experiment (UK)

Last

birthday

Next

birthday

Compliant 37% 44%

Non-compliant 36% 32%

Not known 28% 24%

N (2+ households) 98 100

2013 & 2014 Community Life experimentsLast

birthday

Last/Next

birthday

Non-compliant 25% 25%

Base: All completes (2013=1,689; 2014=10,157)XXX 25% XXX 21%



(2) Random selection from online completed roster

2012 ESS Mixed Mode Experiment (UK)Last

birthday

Next

birthday

37% 44%

36% 32%

28% 24%

98 100

Online random

selection

Compliant 29%

Non-compliant 14%

Not known 57%

N (2+ households) 216



Avoids selection bias

Community Life Survey uses this method

– Up to 4 login details in letter (more provided on request)

– Some evidence that conditional incentive encourages people to

complete multiple questionnaires

– 4% of addresses fill in the questionnaire for more adults than listed

(3) All eligible adults

Sources: TNS BMRB (2015). Active People Survey Methodological Development Summary Report.

TNS BMRB (2013). Community Life Survey: Summary of web experiments.



Respondent selection only required in 15% of households

Motivation to complete multiple questionnaires for incentives is reduced

Small loss of precision due to clustering

– But gain in precision by having less variable selection weights

Active Lives Survey uses this method

– 2 login details printed on letters

(4) Up to 2 eligible adults

Source: Ipsos MORI (2017). Active Lives Survey: Year 1 Technical Report.



Experiment in UK and 16 other EU member states

3 conditions:

1. Letter with login details for up to 2 (or 3) adults

2. Letter requests any adult member to complete web questionnaire;

And if more than one adult in the household:

a) the household decides who the other adult(s) should be

b) programme randomly selects 1 (or 2) other eligible adult

Results not yet available

Fundamental Rights Survey experiment

Source: Fundamental Rights Pilot Survey, 2017


Further research

Assess respondent fraud with the “up to 2”

method

Which method is better with regard to:

– Address-level response rate

– Total number of completed questionnaires

– Sample composition


20

Paste co-

brand

logo here

2. How to encourage

selected respondents to

go online and complete

a questionnaire?


Push-to-web response rates

Highly variable online response rates

– Ranging from about 7% to 25%

– Survey topic & sponsor likely to be important factors

Need to understand the process of response in order to find

effective methods for increasing response rate

Source: Williams, J. (2017). An introduction to address-based online surveying.


Framework for push-to-web response

Motivated to

read the

mailing

Motivated to

go online

Motivated to

take part in

survey

Motivated to

open the

mailing

Motivated to

complete

questionnaire

Personalisation

Type of mailing;

e.g. envelope with

letter, postcard

Appearance of

mailing before

opening; e.g. logo

Personalisation

Easy to read; e.g.

length, font,

vocabulary

Appearance; e.g.

important,

professional

Clarity about the

purpose of the

mailing

Clarity about the

survey request

Use persuasive

reasons for taking

part

Clear instructions

for logging into

the questionnaire

Minimal effort

needed to enter

login details

Multiple access

methods; e.g. any

internet-enabled

device

Use multiple mailings:

• An optimum number of mailings

• An optimum length of time between mailings

• A diverse and yet coherent package of multiple mailings

Landing page

looks authentic

with clear

instructions

Design for mobile;

e.g. short, reduce

clutter and text

Avoid question

types that are

prone to break-

offs

Reduce cognitive

burden


Fairly confident that the following has a positive impact

Motivated to open and read mailing

– Authoritative logos

Motivated to take part in survey

– Multiple and varied contact attempts

– Use of incentives

Motivated to go online and complete

web questionnaire

– Avoid use of complex question formats

– Avoid cognitively difficult questions

– Mobile friendly


Further research

Open the mailing

– Type and appearance of mailing

Selected respondent(s) motivated to take part in survey

– Respondent selection method

– Type of incentive (incl. cost effectiveness)

– Persuasive messages

Selected respondent(s) motivated to go online

– Reducing the effort of moving from postal contact to online data collection

The optimal package of multiple contact attempts

– Dillman (ESRA 2017): “Too often surveyors … ignore the operational connectivity

among them that is essential”


25

Paste co-

brand

logo here

3. How to boost

overall response

rates?


Web non-respondents:

1. Off-liners

2. On-liners who do

not complete the

web questionnaire


Household internet penetration

Source: ONS, Opinions and Lifestyle Survey, Jan/Feb/Apr 2017

10%of UK adults live in households without any access to the internet


But internet penetration is increasing

Over half of 75-plussers (53%)

now have internet access at

home

Source: Ofcom Technology Tracker, Jan-Feb 2017

Among the older age groups


An option for panels; e.g. CRONOS

Too costly for cross-sectional

surveys

How to include offliners?

Provide online device & internet access


How to include offliners & other web non-respondents?

Offer alternative data collection mode

Face-to-face

interview

e.g. FRS Pilot

Telephone

interview

e.g. NatCen panel

Postal

questionnaire

e.g. Community Life

Survey, Active Lives

Survey


Further research

Concurrent or sequential?

If sequential, should the alternative mode be

mentioned upfront?

How to offer alternative mode?

– All web non-respondents

– Sub-sample of web non-respondents

– At request only

Impact of non-response bias on different type

of estimates?


32

Paste co-

brand

logo here

4. How to design questionnaires

that produce comparable data

across modes and devices?


Measurement differences across modes

Interviewer-administered vs self-administered

Visual vs aural presentation

Question wording & format

–Unified mode construction: use same wording and visual

layout of questions (Dillman et al, 2015)

Causes


Measurement differences

Source: Ofcom Technology Tracker, H1 2017

The smartphone is

the most important

device for internet

access.

across devices


Measurement differences across devices

Screen size

Burden of completing questionnaire on mobile

Greater potential for distraction on mobile

Causes

Mobile-first design

Short questionnaires, minimise text length of questions and

answers, keep response lists short, reduce cognitive burden,

etc…


Mobile-first design

Perceived as a constraint on questionnaire design

But it improves focus and discipline

Back to basic principles of

good questionnaire design!


How to transition to push-to-web?

Existing interviewer-administered surveys:

1. Reduction and/or modularisation of long questionnaires

2. Redesign questions that rely on interviewer-administration

– Device-agnostic Mobile-first

3. Risk to time series

a) Accept the break in time series (e.g. Active Lives Survey)

b) Parallel run of old and new methodology (e.g. Community Life Survey)


Further research

Separation of selection & measurement effects

and their impact on different type of estimates

Transitioning from face-to-face to push-to-web

Optimising questions for mobile completion: e.g.

grids, scale length

Modularisation of questionnaires


39

Paste co-

brand

logo here

Where do we go from

here?


Where do we go from here?

• Push to web will increase in use in UK and will replace

some f2f surveys - push and pull factors will continue

• Will this matter? Only yes if less able to draw valid and

reliable conclusions from data. Depends on:1. whether push-to-web data quality inferior

2. how we use the data.


Data quality

• Push-to-web samples demographically worse than face-to-face ones

• For range of survey variables different estimates for P2W and F2F

• Not yet clear (i) why this happens or (ii) which method produces better

survey estimates

• To date UK statistical evidence muddy and relates to a small number of

variables

• If push-to-web surveys replace face-to-face surveys we will see changes in

data but unclear how much they represent a decline in quality (if at all)


How we use the data

• Even if the quality of data does decline, does it matter?

• Often care about change; change measures compatible with inaccurate single-point

estimates if biases hold constant

• Assumption that bias is time-invariant in push to web surveys is heroic given current

state of knowledge

Push-to-web methods will increase - incumbent on us to investigate urgently impact of

this on quality and fitness for purpose:

– Differences in estimates relative to other methods – by type of vartiable

– Causes of these differences – mode effects v. sample composition

– Stability of bias over time - reliability of change estimates


Some useful open access resources:• Don. A. Dillman. (2017). The promise and challenge of pushing respondents to the

Web in mixed mode surveys. Survey Methodology, 43(1): 3-30.

• Williams, J. (2017). An introduction to address-based online surveying. Social

Research Practice, 3: 23-36.

• Community Life Survey: experimental online survey findings. Three reports at

https://www.gov.uk/government/publications/community-life-survey-

experimental-online-survey-findings

• Humphrey, A. & Agur, M. (2013). European Social Survey Round 6 UK Mixed Mode

Experiment. NatCen Social Research.

• Ipsos MORI (2017). Active Lives Survey: Year 1 Technical Report.

https://www.gov.uk/government/publications/community-life-survey-experimental-online-survey-findings


Thank you.

The development of push-to-web surveys in the UK · –2015 Japanese Census –2016 Canadian and...

Documents

Transcript of The development of push-to-web surveys in the UK · –2015 Japanese Census –2016 Canadian and...