Prof.dr. Branko Kontić, dr. Davor Kontić CIVITAS Elan Study Tour Ljubljana, 23 October 2013 Role...
-
Upload
hilary-neal -
Category
Documents
-
view
229 -
download
0
Transcript of Prof.dr. Branko Kontić, dr. Davor Kontić CIVITAS Elan Study Tour Ljubljana, 23 October 2013 Role...
prof.dr.prof.dr.Branko Kontić, Branko Kontić, dr.dr.Davor KontićDavor Kontić
CIVITAS Elan Study CIVITAS Elan Study TourTourLjubljana, 23 October Ljubljana, 23 October 20132013
prof.dr.prof.dr.Branko Kontić, Branko Kontić, dr.dr.Davor KontićDavor Kontić
CIVITAS Elan Study CIVITAS Elan Study TourTourLjubljana, 23 October Ljubljana, 23 October 20132013
Role of evaluation Role of evaluation in the stage of in the stage of measures and measures and project planningproject planning
Aim of this discussionAim of this discussionAim of this discussionAim of this discussion
Key message to the EC – over VANGUARD (why not POINTER?; expecting synthesis report) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting)
Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, common measures, transport policy development)
Key message to the EC – over VANGUARD (why not POINTER?; expecting synthesis report) – for improving the CIVITAS strategic evaluation system (top responsibility for quality projects and efficiency of budgeting)
Potential for contributing to the benefits of CIVITAS projects at operational level (e.g., city mobility level, measures level, common measures, transport policy development)
CIVITAS FameworkCIVITAS FameworkCIVITAS FameworkCIVITAS Famework
How does EC know that How does EC know that the project will meet the the project will meet the goals?goals?
How does EC know that How does EC know that the project will meet the the project will meet the goals?goals?
DIS
SE
MIN
AT
ION
METEOR
CIVITAS I GUARD
CIVITAS II POINTER
CIVITASPlus
DIS
SE
MIN
AT
ION
METEOR
CIVITAS I GUARD
CIVITAS II POINTER
CIVITASPlus
Impact evaluationImpact evaluation
Process evaluationProcess evaluation
Integrated package Integrated package evaluationevaluation
City level City level evaluationevaluation
Cost-Benefit Cost-Benefit AnalysisAnalysis
CIVITAS Evaluation Objectives
Impact Evaluation Guidelines for defining
evaluation scenarios Common core indicators Guidelines for measurement Guidelines for up-scaling Guidelines for reporting
Process Evaluation Collecting information on
all stages of the measure Feedback and support Factors of success and
unexpected barriers Input for policy
recommendations
Evaluation at Project and City Levels
Cross-site Evaluation Comparing results across sites Assessing transferability Conclusions and recommendations
CIVITAS Measures
Reporting Evaluation Results
CIVITAS Evaluation Objectives
Impact Evaluation Guidelines for defining
evaluation scenarios Common core indicators Guidelines for measurement Guidelines for up-scaling Guidelines for reporting
Process Evaluation Collecting information on
all stages of the measure Feedback and support Factors of success and
unexpected barriers Input for policy
recommendations
Evaluation at Project and City Levels
Cross-site Evaluation Comparing results across sites Assessing transferability Conclusions and recommendations
CIVITAS Measures
Reporting Evaluation Results
Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)
Impact Impact Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)Impact Impact Evaluation Framework Evaluation Framework (by POINTER, 2009)(by POINTER, 2009)
Detailed Measure Descriptions
Effects/Impacts Indicators
Evaluation Plans
Measure or Grouped Measure level Evaluations
Project & City understandings
Cross-site Evaluation and Transferability
POINTER
‘Baseline’
‘Business-as-Usual’
‘After’
Support for Recommendations to other
EUROPEAN CITIES
BAU and CBA BAU and CBA Evaluation Evaluation (by POINTER, 2009)(by POINTER, 2009)BAU and CBA BAU and CBA Evaluation Evaluation (by POINTER, 2009)(by POINTER, 2009)
Process Evaluation FrameworkProcess Evaluation Framework(by POINTER, 2009)(by POINTER, 2009)Process Evaluation FrameworkProcess Evaluation Framework(by POINTER, 2009)(by POINTER, 2009)
Measure Measure level:level:336 336
measuresmeasures(100%)(100%)
City City level:level:
26 cities26 cities
ImplementatiImplementationonPreparationPreparation OperationOperation
Selection Selection
POINTER POINTER SupportSupport
Measure Process Measure Process Evaluation Form Evaluation Form
Process Topics & Process Topics & Issues FormIssues Form
FocusedFocused Measure Measure Process Process
Evaluation Form Evaluation Form
Selection Selection
Measure Measure level:level:
Ca. 120 Ca. 120 measuresmeasures
(Ca. (Ca. 30%) 30%)
Measure Process EvaluationMeasure Process Evaluation
Focussed Measure Process EvaluationFocussed Measure Process Evaluation
Process Process Topics & IssuesTopics & Issues
(by POINTER, (by POINTER, 2009)2009)
Overview - LCA of measures and evaluationOverview - LCA of measures and evaluationOverview - LCA of measures and evaluationOverview - LCA of measures and evaluation
Phases/activities of a measurePhases/activities of a measure Evaluation of measure’s Evaluation of measure’s and project goalsand project goals
Transfer&upgradeTransfer&upgrade
Process evaluationProcess evaluation
planningplanning
implementationimplementation
operationoperationImpact evaluationImpact evaluation
CBACBA
Missing componentsMissing componentsMissing componentsMissing components
Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated)
Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)
Strategic view in the overall concept (not transparent and explicit; expectations to be provided by POINTER and Project Manager, however not clearly presented/formulated)
Loop, feedback, monitoring (possibility for consistent adaptation in terms of project goals – need for additional evaluation approach/understanding/tools in the planning stage of the project)
Missing elements in the planning Missing elements in the planning stage 1/2stage 1/2Missing elements in the planning Missing elements in the planning stage 1/2stage 1/2
Role of the Evaluation Plan• Purpose (holistic view); it is not only a deliverable
• Project management context (consultation); better cooperation needed
• Changes in project implementation (measures’ design, implementation, operation) and links to the evaluation plan
Agreements between MLs and Evaluation Team (objectives, data collection, etc.); see next slide
Performance assessment (PA) of a measure in the context of evaluation; see next slide; PA follows all (internal and external) barriers and drivers, process evaluation follows only outside barriers and drivers
Role of the Evaluation Plan• Purpose (holistic view); it is not only a deliverable
• Project management context (consultation); better cooperation needed
• Changes in project implementation (measures’ design, implementation, operation) and links to the evaluation plan
Agreements between MLs and Evaluation Team (objectives, data collection, etc.); see next slide
Performance assessment (PA) of a measure in the context of evaluation; see next slide; PA follows all (internal and external) barriers and drivers, process evaluation follows only outside barriers and drivers
Missing elements in the planning Missing elements in the planning stage 2/2stage 2/2Missing elements in the planning Missing elements in the planning stage 2/2stage 2/2
Links between evaluation and dissemination: evaluation of dissemination activities, dissemination of evaluation results
Links and preparations to strategic and management role of the evaluation, specifically PA and CBA: Is/was money being reasonably spent? Was it worth doing? What could be done better? Which mistakes we should not repeat? Post “lessons learned” approach is less effective!
Links between evaluation and dissemination: evaluation of dissemination activities, dissemination of evaluation results
Links and preparations to strategic and management role of the evaluation, specifically PA and CBA: Is/was money being reasonably spent? Was it worth doing? What could be done better? Which mistakes we should not repeat? Post “lessons learned” approach is less effective!
A sample of A sample of an an agreement between ML agreement between ML and SEMand SEM
A sample of A sample of an an agreement between ML agreement between ML and SEMand SEM
A sample of the PA sample of the Performance erformance AssessmentAssessment table tableA sample of the PA sample of the Performance erformance AssessmentAssessment table table
Possible improvements Possible improvements 1/31/3Possible improvements Possible improvements 1/31/3
Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly.
Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation.
The planned evaluation procedure does not always work according to plan, it needs to be flexible.
Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis.
There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose.
Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.
Within the hierarchy the areas of responsibility are not defined between measure, city and project level. Especially the role of the measure leader with regard to the evaluation needs to be defined clearly.
Not everyone knows how evaluation works and what is needed. The measure leaders are not experts in evaluation.
The planned evaluation procedure does not always work according to plan, it needs to be flexible.
Ljubljana SEM signed agreements with measure leaders on the collection of data, this is a good tool to clarify the role of the measure leaders, asking them to provide data on a regular basis.
There is a pressure from the PM (Project Management) regarding ticking boxes only without checking the content of the evaluation and its wider/strategic purpose.
Measure leaders need to understand what evaluation is about!! Evaluation should be continuous activity in line with measure planning and implementation.
Possible improvements Possible improvements 2/32/3Possible improvements Possible improvements 2/32/3
Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not.
Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour).
Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.
Process evaluation is important for all measures not only focus ones; it clarifies whether implementation is progressing well or not.
Ljubljana introduced performance assessments from the early beginning. It was defined as an addition to the process evaluation – covers also internal barriers and drivers, and monitors some key components of a ML's work and performance of the measure itself (e.g. management, implementation, evaluation). The results are aggregated in a score (colour score system with comprehensive explanation of the rules for assigning a specific score/colour).
Internal Progress Report (IPR) serves primarily for management purposes; evaluation results loose their strength in the context of this reporting. Transposing performance assessment scores into "traffic light system” of the IPRs should be consistent.
Possible improvements 3/3Possible improvements 3/3Possible improvements 3/3Possible improvements 3/3
Role of the evaluation should be somehow integrated in the project approval procedure by the EC!
Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.
Role of the evaluation should be somehow integrated in the project approval procedure by the EC!
Justification: In the case of ELAN more than a year after official start of the project the evaluators were working with MLs and PEM on "what is to be done in the framework of a certain measure and how will the work/measures be evaluated". The consolidation of DoW in terms of evaluation activities is necessary. This could be formalized by a requirement that a DoW quality check is done by the evaluators too, not only by the project manager and project coordinator.
Improved organization of Improved organization of CIVITAS ElanCIVITAS ElanImproved organization of Improved organization of CIVITAS ElanCIVITAS Elan
European CommissionEuropean CommissionEuropean CommissionEuropean Commission
Political Steering GroupPolitical Steering Group
Project Consortium Project Consortium MeetingMeeting
PartnersPartners Site Site CoordinatorsCoordinators
Evaluation Evaluation DisseminationDissemination Scientific Scientific CoordinatorCoordinator
Project CoordinatorProject Coordinator Project ManagerProject Manager
Project Management GroupProject Management Group
Fin
anci
al &
Ad
min
istr
ativ
e
Fin
anci
al &
Ad
min
istr
ativ
e
Rep
ort
ing
Rep
ort
ing
European CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean CommissionEuropean Commission
Political Steering GroupPolitical Steering GroupPolitical Steering GroupPolitical Steering Group
Project Consortium Project Consortium MeetingMeeting
PartnersPartners
Project Consortium Project Consortium MeetingMeeting
PartnersPartners Site Site CoordinatorsCoordinators
Site Site CoordinatorsCoordinators
Evaluation Evaluation DisseminationDisseminationEvaluation Evaluation DisseminationDissemination Scientific Scientific CoordinatorCoordinator
Scientific Scientific CoordinatorCoordinator
Project CoordinatorProject Coordinator Project ManagerProject Manager
Project Management GroupProject Management Group
Project ManagerProject Manager
Project Management GroupProject Management Group
Fin
anci
al &
Ad
min
istr
ativ
e
Fin
anci
al &
Ad
min
istr
ativ
e
Rep
ort
ing
Rep
ort
ing
ManagerManagerManagerManager ManagerManagerManagerManager ManagerManagerManagerManager
MeasureMeasure
MeasureMeasure
MeasureMeasure
MeasureMeasure
MeasureMeasure
MeasureMeasure
Evaluation Evaluation ManagerManager
Evaluation Evaluation ManagerManager
Evaluation Evaluation ManagerManager
LocalLocal
DisseminationDisseminationManagerManager
DisseminationDisseminationManagerManager
DisseminationDisseminationManagerManager
LocalLocal
Improved organization of Improved organization of CIVITAS ElanCIVITAS ElanImproved organization of Improved organization of CIVITAS ElanCIVITAS Elan