Pathfinder Kick-off Meeting

Research Methodology Introduction, Cardiff

2nd March 2011

Richard Redfern, Steve Tarry

2nd March 2011

Welcome and Introductions from presenters and participants

Content:

• General introduction to the day

• Introduction to the research to be undertaken

• Evaluation overview

• Baseline and contexts for the research

• Project Officers’ role – the monitoring, recording, reviewing and reporting of data and relevant information

General Introduction to the Day:

• A recap on what was covered in day 1:– Study participants, roles and responsibilities– Pivotal role of Pathfinder Officers (POs)

• Aim of day 2:– To present an overview of the planned research – Explain how POs will help in developing the evaluation approach and

populating the study Evaluation Framework – Address any questions or concerns about what is proposed– Begin work on developing some of the tools required

• How the day will run:– Presentation of particular topics associated with the research methodology– Discussion of PO role in particular and any issues arising– Interaction required to ensure POs have confidence in what is required

Hand Out Agenda at This Stage

General Introduction to the Day (cont):

• Agenda: – Before lunch:

• Introduction to the research and overview of evaluation approach– Lunch 12.30 – 13.15– After lunch:

• Specific context for the study, the role of case studies and the work of the POs– Finish at 16.15

• Expected outcome:– insight into what will constitute an appropriate evaluation programme and an

clear understanding of PO role

• Required deliverables:– Tools required to support POs in their particular role

• What will follow on:– Day 3 - Exploration of individual sectors within the study– Meetings with host authorities to being process of engagement

Introduction to Planned Research

• The objectives of the Pathfinder Community Research project are to:– Generate robust evidence of which projects work in reducing greenhouse

gas emissions;– Support the trial of different approaches to community action on climate

change;– Evaluate the different ‘models’ of community based project interventions;– Evaluate the delivery of the pathfinder project programme by the programme

management team; and– Assess the role of Pathfinder Officers in delivering programmes.

• These are to be achieved through the evaluation of 18 case study initiatives spread across 6 Local Authority areas.

• The exact nature and location of the Case Studies is to be determined during this scoping phase of the project– this will define the structure and scope of the research activities undertaken

in subsequent phases of the project

Introduction to Planned Research (cont)

• In each of the six Local Authority areas the following outputs are to be achieved by the end of the two year pilot:

– A ‘map’ of communities who are already engaged in action• and those which have potential and/or have demonstrated interest in engaging

in action in the future.

– A ‘map’ of the communities who could be persuaded and encouraged to take action

– A record of carbon reduction work being undertaken by the community

– Specific community groups identified as willing to reduce their carbon footprint

– Baseline data for the carbon footprint of the willing community

– A carbon reduction target of 10 to 20% in the willing community within two years

Research Approach

• The research will look at initiatives within different sectors– Energy– Transport– Water– Food

• Research will look at behaviour changes / effects achieved by initiatives both within and across different sectors

• ‘Theory of Change’ will be used to explore the differences noted and the logic behind what has transpired

• The detailed approach will need to reflect the specific nature of the initiatives that are taken

Participants to think of the nature of the initiatives that might be taken:EnergyTransportWaterFood

Record these; then

Review with results of previous Brainstorm – as summarised on next slides

Possible Initiatives?

Energy· Energy efficient appliances· Energy efficient equipment· Energy efficient lighting· Energy efficient homes· Green buildings and homes· Green business· Green energy· Retrofitting insulation· Programmeable thermostats· Energy conservation

Transport· Anti-idling· Biking· Carpooling· Fuel efficient vehicles· Mass transit· Telecommuting· Vehicle maintenance· Walking

Possible Initiatives?

Water· Drinking water· Greywater· Lawn watering· Low flow showerheads· Low flow toilets· Maintaining wells· Water tanks· Water efficiency· Water restrictions

Food•  Local food·Fertiliser use·Reducing meat and dairy consumption·Biodiversity·Forestry·Pesticide use·Organic food·Reduce packaging

Theory of Change

• Robust framework for considering:• What has changed ?• Why ? • In what context ?

• Tailored to address complex interventions, packages or programmes

• ‘Mapping’ produced to record and test underlying logic of investment:• Links between investment and short term outcomes• Hypotheses of links between short and long term behavioural

change• Work with stakeholders to review mapping and recognise their role

within individual ‘blocks’ and in relation to specific ‘links’

Logic Mapping – the Theory:

What is invested, e.g. money, skills, people, activities

What is invested, e.g. money, skills, people, activities

What has been produced What has been

produced

Short and medium termresults

Short and medium termresults

Long-term outcomes Long-term

outcomes

InputInput OutputOutput OutcomesOutcomes ImpactImpact

The frame-work within which an intervention is

located

The frame-work within which an intervention is

located

Context Context

Analysis of context Analysis of

context Analysis of objectives Analysis of

objectives

Analysis of the logic of the intervention Analysis of the logic of the intervention

Logic Mapping – an Example

Activities OutputsShort-term outcomes

Accredited schemesaudited &

standards maintained

% pupils recruited,completed training

and in receipt of awards

Intermediate outcomes Long-term outcomes

Increased no of school cycling clubs,

networks, & use of promotional tools

Reduced duplication,enhanced collaborationand access to funding

in target schools

Cycle ParkingGrant funding for school

cycle parking/travel grants

Links to SchoolIdentify and develop (via signage, lighting,

crossings and calming)existing traffic free routes

between schools and community facilities

Increased no quality & accessibility of

cycle parking at targeted schools

Increased % of bikes parked in availablefacilities each day

Increase in quality & safety

of % of existing routes linking school and

community

Increase usage of enhanced routes

for cycling to school,wider commuting

and leisure

Package of school interventions measures for the CCTS

Inp

uts

S

taffin

g,

Ma

na

ge

me

nt,

Fu

nd

ing

, P

art

ne

rsh

ip,

Co

llab

ora

tion

Enhanced and safercycling environment

Maintain % of town population

currently regularly cycling

Increase % of town population who

are occasional cyclists

Increase % of town population whoare regular cyclists

Cycling increasingly seen as

social norm

Cycling increasingly seen as acceptable for transport& leisure in school setting

Increase % of those linked to school

setting cycling for leisure& non-work commuting

Increase % of school population

(pupils & staff)commuting to/from

school by cycle

Increase % of town population

cycling from X to Xby 2012

Longer term impacts Modal shift

Reduced congestionReduced pollution

Enhanced environmentReduced Accidents?

Enhanced safety?Increased Physical Activity

Enhanced Health

Increased knowledgeof benefits of cycling and

how to promote it (in teachers parents, pupils)

Increased no (and effectiveness)of school interventions resulting

in increased participation in these by target groups

% of authorities/institutions recruited,

and training accredited

Promotional programmes (targeted at parents, teacher and pupils),

curriculum programmes, travel plans, safety

and training delivered in X% schools/classes.Increased use of good

practice guides

School Champions &Active Recreation

START school championscoordinating school

interventions expanding network of school cycle clubs.developing and implementing

school promotional tools

Increased membership ofand participation

in cycling as part ofschools clubs &

networks &tailored interventions

BikeabilityImplementation of

training (and related certificates) for bike handling

and safe on/off road cycling (3 age related levels)

Increased knowledge & skills& confidence for bike handling and for on and off road cycling

for x% of three age groups(Up to 9, 10-11, secondary

school aged) in CCTs

Bike ItBike It officers

working with school clusters (educating pupils,

teachers, parents, authoritiescoordinating interventions,

curriculum programme, addressing safety,

embedding good practice)

Hand out Spreadsheet example for capturing appropriate data from which to develop appropriate Logic Map

Practical Example - Specific to the Study

Activities Short Term Outcomes Medium Term Outcomes Long Term Outcomes

Cycle Training for Year 5/6

Children

More positive attitude to cycling by

parents/children

Increased cycle skills

Increased risk awareness

Increased propensity for parents to let children cycle

Increased cycling to school

Reduced accidents

involving child cyclists

Reduced carbon emissions

Cycle Training for Year 5/6

Children

More positive attitude to cycling by

parents/children

Increased cycle skills

Increased risk awareness

Increased propensity for parents to let children cycle

Increased cycling to school

Reduced accidents

involving child cyclists

Reduced carbon emissions

Poorly delivered trainingCourse not well designed

Low take-up rateMisses ‘at risk’ children

Role of Case Studies

• Case Studies will give us practical examples of Inputs and Outputs

• Inputs are the investment, resources and processes applied in the implementation of case study interventions, including Local Authority and other stakeholders’ time and investment. Accurately monitoring, analysing and reporting the full range of inputs will be a primary focus for the evaluation framework

• Outputs are the initiatives delivered, as a result of the inputs directed towards the Pathfinder programme. These will cover the range of intervention types (to be defined), including transport, energy, water and food

Categorisation of Initiatives

• Categorisation of initiatives:– provides a focus for data collection– assists when reviewing the transferability of findings about

similar types of interventions.

• Typology of initiatives according to, for example: – Primary objective – Primary target population – Scheme ownership – Geographical scale

.

The Research Programme • Short term

– Engagement– Identify possible Case Studies– Identify data sources– Set up evaluation processes– Identify Indicators of success (and possibly failure)

• Medium term– Data collection– Look at inputs and outputs– Review evaluation approach

• Longer term– Interim

• Look at outcomes and impacts• Review research approach

– Final• Look at outcomes and impacts• Address specific research questions

Q and A

Evaluation Overview - What is Evaluation?

• “Evaluation is the post implementation assessment” of:• Programmes• Policies and strategies• Investment ‘packages’• Individual schemes/interventions

• Two key areas in evaluation:• Outcome or Impact evaluation• Process evaluation

• Both apply to the Pathfinder Research

Evaluation within the Programme / Project Lifecycle

Rationale

ObjectivesFeedback

AppraisalEvaluation

Monitoring

Implementation

Benefits of Evaluating

Benefit Detailed Consideration

Planning and Policy Setting

• Improve carbon reduction plans and policies • Improve decision-making for policy makers and communities• Identify benefits and impacts of individual schemes

Implementation

• Improve the efficiency of carbon management interventions• Better policy delivery across sectors (transport, water, energy)• Develop delivery processes for businesses and community groups• Target engagement approaches and activities

Institutional Strengthening

• Improve co-operation of partners• Promote inter-partner working and carbon management

Accountability

• Evidence and justification for investment (emission reductions)• Ex-post evaluation of value for money and effectiveness• Carbon trading and management outcomes

Knowledge Production

• Identify what has worked well in different circumstances and why• Contribute to the ongoing learning process• Develop and refine project development procedures• Learn lessons

Relationship to Logic Mapping

Impacts

OutputsInputsObjectives

Outcomes

Rationale

• Rationale – Policy context

• Objectives – Local priorities for investment

• Inputs – capital investment and human resources

• Outputs – delivered interventions

• Outcomes – changes in key indicators

• Impacts – longer term changes

Traditional realm of monitoring

Realm of evaluation

Phases of Delivery

Phase 1: Evaluation Scoping

Phase 2: Interim Evaluation

Phase 3: Outcome Evaluation

This phase defines the specific focus of the evaluation and, importantly, the

interventions to be included. It includes the development of the evaluation

framework and its constituent parts.

This phase will identify existing evidence gaps and design data collection activities

to fill these. Initial stakeholder and community engagement will be

undertaken and process evaluation work will be undertaken.

Ongoing data collection and detailed process evaluation activities undertaken

during and after the implementation phase of measures. This will include

further engagement and action research activities.

Phase One (Scoping) – Overview

Evaluation Scoping

Define interventions and packages

Confirm evaluation objectives

Select research questions

Select evaluation indicators

Select core indicators

Select contextual indicators

Stakeholder liaison Conduct ToC mapping

Identify new data needed

Define evaluation approach

ToC mapping

Objectives of the Research

• The objectives of the Pathfinder Community Research project can be grouped around three general themes:

• Outcomes:– Generate robust evidence of which projects work in reducing greenhouse

gas emissions;

• Engagement:– Support the trial of different approaches to community action on climate

change;– Evaluate the different ‘models’ of community based project interventions;

• Processes:– Evaluate the delivery of the pathfinder project programme by the programme

management team; – Assess the role of Pathfinder Officers in delivering programmes.

• The nature of research questions reflect the specific theme under consideration

Participants to think about the nature of questions / hypotheses under each objective on preceding slide

Record; then

Compare with what is on the following slides

Research Questions

• Objective 1: To generate robust evidence of which projects work in reducing greenhouse gas emissions

• Questions in terms of impacts:

· What levels of greenhouse gas emission reductions have been achieved?

· What are the key techniques/factors/activities that have reduced green house gas emissions?

· How do the reductions vary between investment themes?

· What are the timescales for outturn reductions?

Research Questions

• Objective 1:

• Questions in terms of behavioural change contributing to outcomes:

· What changes in behaviour have been generated, how and why? Are these sustainable?

· What are the key drivers and motivators for behavioural change? Do they vary in the different themes? Do these vary between locations and population sub-groups?

· Which approaches and techniques have generated the highest levels of behavioural change?

· To what extent has investment generated low-carbon communities?

· How was the long term sustainability of the project planned for and secured?

Research Questions

• Objective 2: To support the trial of different approaches to community action on climate change

• Questions in terms of community engagement:

· What approaches have been adopted to engender community action?

· Which approaches achieve greatest community engagement and support and why?

· Which project theme generated the greatest level of community engagement and why?

Research Questions

• Objective 2:

• Questions in terms of penetration and sustainability:

· How does the delivery of impacts of different approaches vary between locations, across socio-economic groups and why?

· What are the key factors in a given approach that accelerated success?

· What are the key factors that resulted in less than expected good results?

Research Questions

• Objective 3: To evaluate the different ‘models’ of community based project interventions

· What are the different models/approaches to community based project development?

· Which models/approaches are the most efficient in terms of costs/benefits?

· What are the key processes involved in each model/approach and what lessons can be transferred to other locations/projects?

· Which elements of approaches are most effective/important in reducing greenhouse gas emissions and why?

· Which approaches generate the best advancement in community knowledge and skills in addressing climate change? How do these vary between locations and case studies?

Research Questions

• Objective 4: To evaluate the delivery of the pathfinder project programme by the programme management team

· What co-ordination/leadership approaches by the programme management team facilitated dialogue and collaboration with key stakeholders and between Pathfinder Officers (POs)?

· What approaches acted as barriers to meaningful dialogue and collaboration?

· What support, training and direction provided for clarity of work by the POs?

· How effective was the Advisory Group in helping to deliver the work of the POs?

· What key strengths of the programme management team enabled action by the POs and what gaps/weaknesses acted as barriers?

· What other barriers challenged work of the whole programme?

Research Questions

• Objective 5: To assess the role of Pathfinder Officers in delivering programmes

· What role did the PO have in accelerating the work of the community projects they supported?

· What particular skills, knowledge and abilities of the PO supported the success of the community projects?

· What approaches to community participation taken by the individual POs supported community projects?

· What barriers did the POs face that acted as a barrier to the delivery of the work of the community project? How were they removed, if removable?

Phase One (Scoping) – Overview

Evaluation Scoping

Define interventions and packages

Confirm evaluation objectives

Select research questions

Select evaluation indicators

Select core indicators

Select contextual indicators

Stakeholder liaison Conduct ToC mapping

Identify new data needed

Define evaluation approach

ToC mapping

Participants to consider what would make a Good Indicator

Record, then

Discuss individual merits

What Would Make a Good Indicator

• SMART:– Specific– Measurable– Achievable– Relevant– Time based

What Would Make a Good Indicator (cont)

• But, what will be Practical?:– What data are available?– Who owns or has access to the data?– What are the timescales for anticipated change?

Q and A

Lunch

Baseline and Contexts

Top Down Policy – Bottom up Community Actions

Current Baseline Emissions

FutureTarget

Policy

Act

ions

What Differences are Made as a Result of Pathfinder Officer Actions?

Need to Look Back as well as Record the Future (for lessons to be laerned) No PO Involvement

PO Engagement? PO Engagement?

Project Timeline

Problem Concept Development Implementation Outcome Impact

Project Lifecycle

Time

Impact of PO

Project might progress at a faster pace – with impact felt sooner

How to assess what would have happened without engagement?

Informed by ‘backward’ review and comparison with other similar projects without PO engagement

What Differences are Made as a Result of Pathfinder Officer Actions?

Need to Look Back as well as Record the Future (for lessons to be laerned)

PO Engagement? PO Engagement?

Scale of Impact No PO Involvement

Level of Community Engagement / Take-up

Impact of PO

Overall impact of Project might increase

The availability of data for reviewing the baseline / activities prior to PO engagement will be a factor in the selection of case Studies

Q and A

Phase One (Scoping) – Overview

Evaluation Scoping

Define interventions and packages

Confirm evaluation objectives

Select research questions

Select evaluation indicators

Select core indicators

Select contextual indicators

Stakeholder liaison Conduct ToC mapping

Identify new data needed

Define evaluation approach

ToC mapping

Monitoring and Recording

• Following initial engagement with Stakeholders– Focus will be on collating basic ‘headline’ data for each potential case study

(within a long list of, say, 120)– This will enable actual case studies to be selected for evaluation

• Criteria to be determined to facilitate the selection of final case studies – Criteria should allow comparisons to be made across candidate case studies– Should aim for a representative cross section of case studies, covering all

relevant sectors and differing approaches to community involvement, project development and delivery

• More detailed data will then to be recorded for selected case studies, to fully define and characterise the case study in advance of the evaluation phase

• The actual basic data to be collated will reflect the nature of the initiative and the criteria used for choosing between candidate case studies

Participants to brainstorm some of the criteria / data needs

Record

Handout initial draft schema for reviewing projects (seeking financial support under sustainable communities programme)

Participants to take this away and provide feedback on how this can be adapted to the needs of the Pathfinder research

Q and A

Evaluation is Not Just About the Data

• Data

• Facts

• Information

• Intelligence

Insight

Journal to be kept :

For each activity undertaken:

Describe the activity Reflect on: what went well what didn’t go well

and why notwhat should be done

differently next time

Note down progress and additional information as required

Capturing the Data (Facts, Information, Intelligence and Insights)

• Precise Tools to be Developed

• But, likely use of dedicated site to store information and share this across the study team

• Possible input from Development Officers in designing appropriate data capture proforma

Hand out example field sheet

Participants to review this and feedback later how this might be adapted - to meet their needs / needs of the study team

Wrap-up

• Future Actions

Q and A

Thank You

richard.redfern@aecom.com

steve.tarry@aecom.com