Evaluating behaviour change programs - WasteMINZ · PDF fileEvaluating behaviour change...

Click here to load reader

  • date post

    08-Sep-2018
  • Category

    Documents

  • view

    217
  • download

    0

Embed Size (px)

Transcript of Evaluating behaviour change programs - WasteMINZ · PDF fileEvaluating behaviour change...

  • Evaluating

    behaviour change

    programsprograms

    Liz Ampt

    Concepts of Change

  • What are we measuring?

    - Whether people are

    doing things differently

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    doing things differently

  • How can we measure it?

    - Measuring changes in levels of reducing/ reusing/

    recycling

    - Observing/recording behaviours

    - Asking about change

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    - Each is a form of survey or data collection exercise

  • SampleDesign

    PilotSurvey

    Conductof

    Selection ofSurvey Method

    SurveyInstrument

    Design

    PreliminaryPlanning

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    ofSurvey

    DataCoding Data

    Editing

    Data Managementand Analysis

    Data Correctionand Expansion

    Presentationof Results

    Tidying-Up

    The survey

    process

    Richardson, Ampt,

    Meyburg 1995

  • Each is a form of survey Method of measurement Aspects of survey design needed

    Measuring changes in levels of

    reducing/ reusing/ recycling

    - Sample selection

    - Pilot

    - Survey

    - Expansion/weighting

    - Analysis

    Observing/recording behaviours - Sample selection

    - Pilot

    - Survey

    - Expansion/weighting

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    - Expansion/weighting

    - Analysis

    Asking about change - Sample selection

    - Survey design

    - Pilot

    - Survey

    - Expansion/weighting

    - Analysis

  • Key elements of survey design

    1. Preliminary planning

    2. Selection of method

    3. Sample design

    4. Survey instrument design

    5. Pilot

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    5. Pilot

    6. Survey implementation

    7. Expansion/weighting

    Analysis over to you..

  • 1. Preliminary planning

    Define survey objectives

    Very specific: what, by whom, over what period, where

    Review of existing information

    Useful methodologies from elsewhere; use of stated

    preference?

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    preference?

    Define terms

    From your objectives and from respondents perspective

    Survey content

    Dot points

    7

  • Exercise

    Think of 2 terms you would want to use in a survey

    related to your work

    Write down clear definitions for both

    Ask another person (not from your organisation) to do the

    same with your terms

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Compare

  • 2. Selection of a Survey Method for Measurement

    Observation surveys

    Intercept surveys

    Self-administered surveys

    Telephone surveys

    Personal interview surveys

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Personal interview surveys

    Internet/online surveys

    9

  • Observation methods

    Chosen when

    Possible to count accurately

    Possible to count all or select a representative sample

    Can be manual, automatic, video

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014 10

  • Intercept Surveys

    Intercepting people

    At an activity centre (e.g. workplace, transfer station, shopping centre)

    Possible methods

    distribution - mail-back/on-line

    personal interview

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    collect phone no.

    Always needs a total classification count

    11

  • Intercept Surveys

    Advantages

    Able to reach specific populations

    Can combine with observational counts

    Can use multiple survey methods

    Disadvantages

    Generally low response rates (20-30% for self-completion)

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Generally low response rates (20-30% for self-completion)

    Hurried conditions

    Must allow for non-random sampling

    No follow-up possible in most cases

    12

  • Self-administered Surveys

    Possible targets

    households

    activity centres/workplaces/transfer stations

    Method of Distribution

    mail-out vs. hand delivered

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Method of Collection

    mail-back vs. hand collection

    13

  • Self-administered Surveys

    Advantages

    Can get extensive geographical coverage

    No interviewer effects

    Can obtain considered responses

    Hand-collection to good response rate

    Disadvantages

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Disadvantages

    Layout and wording must be clear - hard to design

    No probing possible

    Answers not independent

    Response rates lower than face to face

    14

  • Telephone Interviews

    Advantages

    Wide geographic coverage

    Intermediate costs

    Good supervision - CATI

    Multilingual capabilities

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Computerised

  • Telephone Interviews

    Disadvantages

    Sample usually weak

    Low phone ownership for some groups

    Answer-phones, mobile phones, screening devices

    Hard to know how it represents the population

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Hard to know how it represents the population

    Credibility of interviewer (confusion with telemarketing)

    Low response rate

    No follow-up for refusals

  • Personal Interviews

    Can be paper or computer

    Advantages

    Generally higher response rates (60-90%)

    Flexibility of information

    Presence of interviewer

    Maintain interest

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Maintain interest

    Spontaneous answers

  • Personal Interview

    Disadvantages

    High costs

    Interviewer influence

    personal characteristics

    interrupt household/work routine

    opinions of interviewers

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    opinions of interviewers

    interpretation of vague answers

    Considered response difficult

  • On-line Surveys

    Advantages

    Low costs

    Can use elaborate visual effects

    Can use adaptive techniques (can give different scenarios

    for different responses)

    Good for workplaces if sufficient follow-up

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Good for workplaces if sufficient follow-up

    Disadvantages

    Usually very biased sample

    Low response rate

    Hard to get all people in household if needed

  • Exercise

    Think of a behaviour you would like to measure

    Discuss with a partner

    Best method of collection

    Strengths

    Weaknesses

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Weaknesses

  • 3. Sample Design in the Survey Process

    SampleDesign

    PilotSurvey

    Conductof

    Survey

    Selection ofSurvey Method

    SurveyInstrument

    Design

    PreliminaryPlanning

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    DataCoding Data

    Editing

    Data Managementand Analysis

    Data Correctionand Expansion

    Presentationof Results

    Tidying-Up

    Sampling Methods

  • Preliminary Concepts

    What is a sample?

    a collection of things which is some part of a larger population and which is selected so as to be representative of some or all of that population

    Target Population

    who are we trying to survey?

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    Sampling Units

    what are we going to sample?

    Sampling Frame

    where are we going to get a list of these things?

    Sampling Methods

  • Sampling Frame

    a base list to identify the sampling units

    should contain all the sampling units

    examples,

    all households on a street (e.g. Council records)

    telephone directories

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    telephone directories

    mailing lists

    maps

    electoral rolls

    blocklists

    Sampling Methods

  • Sampling Frame Problems

    inaccuracy

    incompleteness

    duplication

    inadequacy

    out-of-date

    WasteMinz Roundup 2014was WasteMINZ Roundup 2014

    out-of-date

    Must check the reason for which the list was

    originally compiled to understand likely deficiencies.

    Sampling Methods

  • Sampling Error & Sampling Bias

    Samplin