Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive...
description
Transcript of Long term evaluation – summarizing “How-to-do-it” CIVITAS Thematic Groups, Interactive...
Long term evaluation – summarizing “How-to-do-it”
CIVITAS Thematic Groups, Interactive training
7 May 2014, ECOMM 2014, Florence, Italy
Anders Wretstrand, Lund University
22
« How-to-do » long term evaluation
Each measure
• has one or more objectives
• may be a combination, a bundle, of measures
Objectives may be like
• Modal shift
• Vivid city centre
• Coordinated urban freight
These have to be operationalized into indicators that are
• Linked to objectives
• Quantifiable
• Measurable
• Each objective relate to several measures
• Each measure should be linked to several indicators
33
Impact and process evaluation
Impact evaluation
• is carried out in several steps
• starting with a baseline study (before)
• after implementation (after - short term, 1st or 2nd year)
• after “a while” (after - long term, 5th or 10th year)
• what matters is what impacts the measure has on the particular area that the measure aims at, revealed by monitoring indicators
Process evaluation
• a means to facilitate dissemination and transferability of the results
• tells the story of
• planning and design
• implementation and operation
• reveals the “road to success” (or failure)
44
Short and long term
Short term evaluation
• generally covers one year or year or duration of project funding duration
• uses before and after data
Long term evaluation
• can include projecting impacts into future
• Forecasting, scenario-building
• can involve time series data for programs/measures over years
• running, ongoing surveys
• planned, repeated long-term effect surveys
Different results…
• no impact evidence on short term – but positive on long term
• impact on short term – but no “up-scaling” on long term
• in both cases: the process (the story) is important to capture!
55
About effects – impact assessment
Measure
Before After - Short term
TimeTime
Impact
Impact
Effect of measure
Long termLong term
Beware!
66
About effects – impact assessment
Measure
Before After - Short term
TimeTime
Impact
Impact
Effect of other factors
Effect of measure
Long termLong term
”Business as usual”
77
Evaluation design options
Before MeasureMeasure After I
Measure site
Baseline Impact I
After II
Impact II
Before After I
Control site
After II
Changes?Changes?
Changes?Changes?
The ideal situation with control site – nice but quite rare. To strive for!
88
Evaluation design options
Before MeasureMeasure After I
Measure site
Baseline Impact I
After II
Impact II
After I
Control site
After II
Changes?Changes?
Changes?Changes?
Control site without baseline
99
Evaluation design options
MeasureMeasure After I
Measure site
Impact I
After II
Impact II
After I
Control site
After II
Changes?Changes?
Changes?Changes?
No baseline data
1010
Evaluation design options
Before MeasureMeasure After I
Measure site
Baseline Impact I
After II
Impact II
Changes?Changes?
No control site to compare with
1111
Evaluation design options
MeasureMeasure After I
Measure site
Impact I
After II
Impact II
Changes?Changes?
No control site or baseline data to compare with
Common situation when asked to evaluate!
1212
Evaluation design options
MeasureMeasure
Measure site
After
Impact
Changes?Changes?
No control site, previous evaluation or baseline data to compare with
Common situation when asked to do long-term evaluation!
1313
Examples – measures from DYN@MO
Cases
• planning: SUMP
• health and accessibility: healthy walking
• public transport: more efficient trolley buses
• mobilit management in residential areas
1414
Examples – SUMP
Measure Objective Indicator Method Baseline
SUMP To (plan to) implement more “sustainable” transport measures than previously
“Sustainability” of measures previously implemented and in new SUMP; and those from SUMP that are actually implemented.
Score using checklist methodology (e.g. Ecomobility SHIFT, or a simplified version)
Analysis of recently implemented measures,
One of the objectives and indicators
Very suitable for long term evaluation!
1515
Examples – Planning for healthy walking
Measure Objective Indicator Method Baseline
Healthy walking
Improve health especially amongst older people in the CIVITAS area
Levels of physical activity that target group in area undertake – self-reported
Questionnaire to 100-200 patients at each Centro de Salud; and a panel of the same 30 people each year.
First survey results before implemen-tation
One of the objectives and indicators
Very suitable for long term evaluation! But still……difficult to assess long term effects, since people age!!!
1616
Examples – More efficient trolley buses
Measure Objective Indicator Method Baseline
”Supercaps” Economic efficiency
Operating and capital costs
Calculations using records and collecting technical data
Existing records and data sets
One of the objectives and indicators
Easy to do short term evaluation as well! Long term evaluation reveals if system e.g. requires more maintenance. Baseline is less of a problem when dealing with technical data!
1717
Examples – Electromobile living
Measure Objective Indicator Method Baseline
Offer shared e-vehicles and pedelecs for tenants
Less car-dependent lifestyle
Travel behaviour, modal split
Case/control, annual travel diary
First survey
One of the objectives and indicators
THEORETICALLY easy to do short term evaluation, but might only detect attitudes and awareness! Long term reveals if people substantially have changed their behaviour, e.g. got rid of 2nd cars
1818
Finally
The « How-to-guide on long term evaluation» from CiViTAS CAPITAL
• will be available for all participants after approval as a final document
• through us or Mobiel21
Now, as a hand-out
• Checklist: recommendations for long term evaluation of MM activities
• If you or anyone else have done a previous evaluation of the measure, or
• If no previous evaluation exists
1919
Checklist: based on existing short-term evaluation
Steps RemarksImpact evaluation
Identify policy/measure objectives Break down the policies/objectives into discrete parts
Reflect on cause and effect relations Possible long-term behavioural adaptations and changes?
Select short-term evaluation indicators They should match and measure the discrete objectives
Analyse short-term impact evaluation Good methods to be continued, poor methods to be left out
Revise and refine indicator descriptions Make use of ongoing surveys, revise some indicators if needed
Collect new data Align with short term evaluation to continue the time series
Analyse data Use proper methods for analysis, based on guidelines
Interpret data Are some indicators influenced by other measures? Bundles?
Process evaluation
Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc.
Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned
2020
Checklist: no short-term evaluation available
Steps RemarksImpact evaluation
Identify policy/measure objectives Break down the policies/objectives into discrete parts
Reflect on cause and effect relations Possible long-term behavioural adaptations and changes?
Propose indicators They should match and measure the discrete objectives
Seek for available historical data Which technical and/or survey data sources are available?
Revise and refine indicator descriptions Change of direction may be needed, because of poor data
Collect new and historical data New = current situation; historical = available time series
Analyse data Use proper methods for analysis, based on guidelines
Interpret data Are some indicators influenced by other measures? Bundles?
Process evaluation
Evaluation of existing documents New policies in place, upscaling of measures, new funding, etc.
Self-assessment of stakeholders Interviews, workshops: reported failure, success, lessons learned
2121
Thank you!
Anders Wretstrand
Tom Rye
Contact DetailsContact Details
Lund University
P.O. Box 118, SE-22100 Lund, SWEDEN
http://www.civitas.eu