Evaluating behaviour change programs Liz Ampt Concepts of Change.

47
Evaluating behaviour change programs Liz Ampt Concepts of Change

Transcript of Evaluating behaviour change programs Liz Ampt Concepts of Change.

Evaluating behaviour change programs

Liz AmptConcepts of Change

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

How can we measure it?

- Measuring changes in levels of reducing/ reusing/ recycling

- Observing/recording behaviours- Asking about change

- Each is a form of survey or data collection exercise

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

The survey processRichardson, Ampt, Meyburg 1995

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Each is a form of survey Method of measurement Aspects of survey design needed

Measuring changes in levels of reducing/ reusing/ recycling

- Sample selection- Pilot- Survey- Expansion/weighting- Analysis

Observing/recording behaviours - Sample selection- Pilot- Survey- Expansion/weighting- Analysis

Asking about change - Sample selection- Survey design- Pilot- Survey- Expansion/weighting- Analysis

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Key elements of survey design

1. Preliminary planning2. Selection of method3. Sample design4. Survey instrument design5. Pilot6. Survey implementation7. Expansion/weighting• Analysis – over to you..

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

1. Preliminary planning

• Define survey objectives– Very specific: what, by whom, over what period, where

• Review of existing information– Useful methodologies from elsewhere; use of stated

preference?

• Define terms– From your objectives and from respondent’s perspective

• Survey content– Dot points

7

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Exercise

• Think of 2 terms you would want to use in a survey related to your work– Write down clear definitions for both– Ask another person (not from your organisation) to do the

same with your terms– Compare

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

2. Selection of a Survey Method for Measurement

• Observation surveys• Intercept surveys• Self-administered surveys• Telephone surveys• Personal interview surveys• Internet/online surveys

9

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Observation methods

• Chosen when – Possible to count accurately– Possible to count all or select a representative sample

• Can be manual, automatic, video

10

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Intercept Surveys

• Intercepting people – At an activity centre (e.g. workplace, transfer station,

shopping centre)• Possible methods

– distribution - mail-back/on-line– personal interview– collect phone no.

• Always needs a total classification count

11

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Intercept Surveys

• Advantages– Able to reach specific populations– Can combine with observational counts– Can use multiple survey methods

• Disadvantages– Generally low response rates (20-30% for self-completion)– Hurried conditions– Must allow for non-random sampling– No follow-up possible in most cases

12

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Self-administered Surveys

• Possible targets– households– activity centres/workplaces/transfer stations

• Method of Distribution– mail-out vs. hand delivered

• Method of Collection– mail-back vs. hand collection

13

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Self-administered Surveys

• Advantages– Can get extensive geographical coverage– No interviewer effects– Can obtain considered responses– Hand-collection to good response rate

• Disadvantages– Layout and wording must be clear - hard to design– No probing possible– Answers not independent– Response rates lower than face to face

14

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Telephone Interviews

• Advantages– Wide geographic coverage– Intermediate costs – Good supervision - CATI– Multilingual capabilities– Computerised

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Telephone Interviews

• Disadvantages– Sample usually weak

• Low phone ownership for some groups • Answer-phones, mobile phones, screening devices• Hard to know how it represents the population

– Credibility of interviewer (confusion with telemarketing)– Low response rate– No follow-up for refusals

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Personal Interviews

• Can be paper or computer• Advantages

– Generally higher response rates (60-90%)– Flexibility of information– Presence of interviewer– Maintain interest– Spontaneous answers

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Personal Interview

• Disadvantages– High costs– Interviewer influence

• personal characteristics• interrupt household/work routine• opinions of interviewers• interpretation of vague answers

– Considered response difficult

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

On-line Surveys

• Advantages– Low costs– Can use elaborate visual effects– Can use adaptive techniques (can give different scenarios

for different responses)– Good for workplaces if sufficient follow-up

• Disadvantages– Usually very biased sample– Low response rate– Hard to get all people in household if needed

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

3. Sample Design in the Survey Process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Preliminary Concepts

• What is a sample?– a collection of things which is some part of a larger

population and which is selected so as to be representative of some or all of that population

• Target Population– who are we trying to survey?

• Sampling Units– what are we going to sample?

• Sampling Frame– where are we going to get a list of these things?

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Sampling Frame

• a base list to identify the sampling units• should contain all the sampling units• examples,

– all households on a street (e.g. Council records)– telephone directories– mailing lists– maps– electoral rolls– blocklists

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Sampling Frame Problems

• inaccuracy• incompleteness• duplication• inadequacy• out-of-date

• Must check the reason for which the list was originally compiled to understand likely deficiencies.

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Sampling Error & Sampling Bias

• Sampling Error– due to the simple fact that we are taking a sample, and not

the population. – can minimise error by taking larger sample.

• Sampling Bias– due to systematic omission of some elements from our

final sample.– cannot minimise error by taking larger sample.

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Random Sampling

• Each unit is selected independently and each unit in the population has an equal probability of being selected.

• Must use random sampling to avoid sampling bias.

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Random Sampling Methods

• Simple Random Sampling

• Stratified Random Sampling

• Variable Fraction Stratified Random Sampling

• Multi-stage Sampling

• Cluster Sampling

• Systematic Sampling

• Note quotas not on list

Sampling Methods

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Sample Size

• How much data do we need?

• Too much data >>> too expensive• Not enough data >>> not able to draw conclusions

• Somewhere in the middle is a sample size which enables us to draw sufficient conclusions at a reasonable cost

Sampling Methods

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

4. Survey design in the survey process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Question Content

• Reliability– repeatable– easy to answer

• Accuracy – no question bias– measures what we want

• Relevance– must appear relevant to respondent

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Question Types

• Factual– “What did you do?”

• Classification (e.g. socio-demographic)– for comparing with secondary data

• Opinion and attitude questions– “What do you think about ……?”

• Stated Response Questions– “What would you do if ……?”

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Physical Design of Forms/Apps

• Observational surveys– Ergonomic– Size/format – not too big or small– Weather-proof– Need log forms– Test under actual conditions

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Physical Design of Forms• Self-administered forms

– Layout vital– Minimal writing should be required– No coding aids should appear– Instructions very clear– Professional appearance– Include ID number

35

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

5. Pilots in the survey process

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Pilot Surveys

• Why not do a pilot survey?– too expensive– not enough time

• Why do a pilot survey?– too expensive to omit it– not enough time to omit it

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Uses of the Pilot Survey

• "Skirmishing" of wording• Adequacy of questionnaire

– definitions clear?– too many "don't knows“?– too long?– open to closed questions

• Efficiency of interview/surveyer training• Non-response rate• Analysis• Cost and duration

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

6. Survey implementation

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Conducting the Surveys

• Need high response rate for validity

• Consider– Announcement letter/message >> higher response– Follow-up regime >> higher response rate

Gross sample . . . . . . . . . . . . . . . . . . = 100 (houses who are eligible to put out bin) Sample loss (vacant, invalid phone no.) = 5 (vacant houses) Net sample . . . . . . . . . . . . . . . . . . . . . . = 95 Responses . . . . . . . . . . . . . . . . . . . . . . = 50 (put out green bin) Response rate . . . . . . . . . . . . . . . . . . . . = response/net sample = (50/95) = 53%

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Exercise – dot points for a survey you need

• What method?• How to get a sampling frame?• What questions? Need any for weighting?• Other issues/questions?

– Richardson, Ampt, Meyburg (1995)– http://www.geog.ucsb.edu/~deutsch/geog111_211a/

code_books/Survey_Methods_For_Transport_Planning.pdf

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

7. Weighting (correction)/Expansion

SampleDesign

PilotSurvey

Conductof

Survey

DataCoding Data

Editing

Data Managementand Analysis

Data Correctionand Expansion

Presentationof ResultsTidying-Up

Selection ofSurvey Method

SurveyInstrument

Design

PreliminaryPlanning

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Weighting & Expansion of Data

• Getting the sample data to represent the population from which it was drawn, as nearly as possible

• Why? – systematic errors– Non-response >> weighting certain type of respondents

higher

– Missing Data >> can make assumptions – or note

– Inaccurate Reporting >> e.g. social desirability bias

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Example of weighting

• Your response is 50/95 – what about the 45?Say 30 males (67%) 15 females (33%)

- Your secondary data (e.g. counts, other data)

males 50% females (50%)– missing responses from females

- Responding females are therefore ‘weighted’ with a slightly higher value (1.5) males (.75)

Stopher, P. (2012) Collecting, Managing and Assessing Data Using Sample Surveys , Cambridge.

WasteMinz Roundup 2014was WasteMINZ Roundup 2014

Summary

• To measure behaviour change it is essential to understand the data collection and survey process

• In particular need to understand:– Survey method

– Sampling principles

– Importance of a pilot

– Implementation options

– Need for weighting

• Vital for future funding as well as for sharing methodologies