Evaluation Resource Booklet - Department of Infrastructure ... · Creative Commons licence With the...

Evaluation Resource Booklet2016–21

© Commonwealth of Australia, 2016 ISBN 978-1-925401-44-8 June 2016/INFRA2828

Ownership of intellectual property rights in this publication Unless otherwise noted, copyright (and any other intellectual property rights, if any) in this publication is owned by the Commonwealth of Australia (referred to below as the Commonwealth).

Disclaimer The material contained in this publication is made available on the understanding that the Commonwealth is not providing professional advice, and that users exercise their own skill and care with respect to its use, and seek independent advice if necessary.

The Commonwealth makes no representations or warranties as to the contents or accuracy of the information contained in this publication. To the extent permitted by law, the Commonwealth disclaims liability to any person or organisation in respect of anything done, or omitted to be done, in reliance upon information contained in this publication.

Creative Commons licence With the exception of (a) the Coat of Arms; (b) the Department of Infrastructure’s photos and graphics; copyright in this publication is licensed under a Creative Commons Attribution 3.0 Australia Licence.

Creative Commons Attribution 3.0 Australia Licence is a standard form licence agreement that allows you to copy, communicate and adapt this publication provided that you attribute the work to the Commonwealth and abide by the other licence terms.

A summary of the licence terms is available from http://creativecommons.org/licenses/by/3.0/au/deed.en. The full licence terms are available from http://creativecommons.org/licenses/by/3.0/au/legalcode. This publication should be attributed in the following way: © Commonwealth of Australia 2016, Evaluation Resource Booklet 2016–21.

Use of the Coat of Arms The Department of the Prime Minister and Cabinet sets the terms under which the Coat of Arms is used. Please refer to the Department’s Commonwealth Coat of Arms and Government Branding web page http://www.dpmc.gov.au/guidelines/index.cfm#brand and in particular, the Guidelines on the use of the Commonwealth Coat of Arms publication.

Contact us This publication is available in hard copy or PDF format. All other rights are reserved, including in relation to any Departmental logos or trademarks, which may exist. For enquiries regarding the licence and any use of this publication, please contact:

Director — Publishing and Communications Section Department of Infrastructure and Regional Development GPO Box 594, Canberra ACT 2601 Australia

Email: [email protected] Website: www.infrastructure.gov.au

Evaluation Resource Booklet2016–21

An effective intervention, but not necessarily the most efficient option

It runs really well – but is it taking me where

I want to go?

An efficient intervention, but not necessarily effective

An effective intervention, but not necessarily the most efficient option

It takes me to where I want to go, but is it the best vehicle for the job?

Evaluation Resource Booklet 2016–21

3

Contents

Purpose of this Booklet 4

Evaluation-related activities in the Department 4

Training, workshops, seminars 4Developing a policy/programme/regulatory logic 5Planning for short, medium and long-term outcomes 6Developing a M&E framework 6Undertaking an evaluation or regulatory review 7Management response 10Lessons learned 10

Templates 10

Logic diagram template 11M&E framework and data matrix template 12Evaluation plan template 13Evaluation report template 14Management response template 15Lessons learned template 16

Examples of past evaluation-related activities 17

Reference list 19

Department of Infrastructure and Regional Development

4

Purpose of this BookletThis Booklet accompanies the Department’s Evaluation Strategy 2016–21 and is designed to give practical information and guidance to staff before undertaking an evaluation-related activity:

• an outline of key concepts;

• details of the support available;

• templates for use (and modification as required) when developing a policy/programme/regulatory logic; monitoring and evaluation (M&E) framework; evaluation plan; evaluation report; management response; and a summary of the lessons that were learned from the evaluation or regulatory review;

• examples of evaluation-related activities that the Department has undertaken from 2011–12 to 2015–16;

• a summary of the governance and oversight of the Department’s evaluation-related activities, including relevant committees; and

• a reference list for staff who may want more in-depth information about evaluation and related concepts.

Evaluation-related activities in the DepartmentTraining, workshops, seminarsA range of training, workshops and seminars will be undertaken in the Department, including:

• small-group mentoring sessions with evaluation specialists, primarily in response to demand from line areas;

• mentoring to assist line areas in developing draft policy/programme/regulatory logics; M&E strategies; evaluation plans or draft reports;

• regular training courses that are run in-house, e.g. on how to develop a logic and an M&E framework, and how to manage and conduct an evaluation or regulatory review; and

• occasional all-staff seminars on evaluation-related topics.

The Department’s intranet/ENTR site (under ‘evaluation’) lists upcoming events and contact details for the Governance Unit.


5

Developing a policy/programme/regulatory logicA policy/programme/regulatory logic diagram explains:

• what an activity intends to achieve;

• how and why activities will achieve the intended results;

• the rationale for the causal links between inputs, outputs and outcomes; and

• the assumed outcomes of actions.

It is often represented as a diagram on a page, showing:

• The Portfolio Budget Statement outcome – for the policy/programme/regulation (if available), demonstrating a link to how the activity will contribute to achieving the Department’s purpose;

• The outcomes and contributory outcomes – the changes that will be brought about, and may be expressed as short, medium or long-term outcomes; the overall outcome will normally provide the basis for the objectives;

• The outputs – the tangible product of the inputs and projects, which are fundamental to the outcomes being achieved, e.g. plans, reports, grants and contracts;

• The activities – the main activities (or projects) that turn the inputs into outputs; e.g. processing grant applications;

• The inputs – normally the funds, people and other resources required for the policy/programme/regulation to operate; and

• Context and assumptions – key assumptions that the policy/programme/regulation has been developed on; the context in which it is operating; and activities that may be outside the Department’s direct responsibility.

It is best to work on creating a logic diagram with team members and other relevant areas within the Department. This way, the logic process gathers input from the departmental officers who are involved in the work and ensures you are all on the same page.

A logic diagram identifies the outcomes and subsequent success measures which provide the basis for monitoring, reporting and evaluating progress in meeting objectives over the life of the policy, programme or regulation.

Policy logics ask additional questions, about: the drivers that have prompted government intervention or action, the problem the action was intended to solve, the consequences of failing to address the problem, the barriers that prevent the problem from being resolved, why is it necessary to intervene, what else is underway to help address the problem, do we need to work in partnership with others, what data is available to inform an evaluation.

Logic diagrams are a useful communication tool, which help explain the policy/programme/regulation to stakeholders and clients.


6

Planning for short, medium and long-term outcomesMost managers report on operational measures: contracts signed, hits on websites, units delivered. But is the activity achieving its intended outcomes?

By talking about long and short term measures of success, managers and policy developers can have a conversation about the intention of policy and how to ensure it hits the mark. It‘s often useful to work backwards:

• Success in the long term

What would ‘success‘ look like if the original policy problem was ‘fixed‘? What attitudes and behaviours would need to have changed? Which stakeholders or influencers would have ‘come on board‘? Is there a point where your agency could wind up the program or hand responsibility to others?

• Success in the mid term

Is the activity getting traction? If you are on track to achieve the long term goals, you should see some symptoms of this in the mid-term. For example, there may be a change in attitudes or in the way the issue is reported in the media. Information requests may rise. Key stakeholders may begin taking responsibility for the problem, allocating resources or developing initiatives of their own.

• Success in the first year

So — if you are going to achieve the outcomes you want in the medium and long term, what do you need to do in the first year? Yes, you need to have worked through all the logistics: resources, timelines, infrastructure, contracts, communication and deliverables. But is that enough? To achieve some of the outcomes you need in the mid-term, you may need to show that you have also laid early foundations for important partnerships with stakeholders and considered the linkages with other activities and initiatives.

Developing a M&E frameworkAn M&E framework (or strategy) allows the measurement of the success of a policy/programme/regulatory activity, to know if it is on track to meet its objectives.

The main features of an M&E framework are the:

• Logic diagram – e.g. what are the inputs, outputs and outcomes?

• Measures of success – e.g. what would indicate that you are achieving the outcomes?

• Data sources – e.g. what data and information needs to be collected, and when, to form an evidence base?

• Evaluation strategy – e.g. the approach that will be used to evaluate the policy/programme/regulation over the course of its life, along with possible evaluation questions.


7

Undertaking an evaluation or regulatory reviewEvaluation is the systematic and objective assessment of a policy, programme, or regulatory activity. It normally addresses aspects of efficiency, effectiveness and appropriateness. This leads to well-informed judgements about the impact, achievements and/or merit of that activity.

Evaluation (or regulatory review) findings inform decision-making. Disseminating findings widely allows them to be used to deliver improved infrastructure, transport and regional services; effectively implement government policies, programmes and regulations; and develop investment strategies.

By using evaluation processes consistently in our work, we ensure that:

• current activities are robust, effective and efficient; and are responding to the relevant key issues;

• practical policy options are identified to address emerging issues and challenges;

• skills and capabilities are actively built to achieve the Department’s vision and outcomes; and

• risks are proactively managed at all relevant levels; for the program, Department and government.

The Department’s evaluation activities have strategic oversight by an executive Steering Committee (SC), which endorses the evaluation plans and draft reports. The Department’s Finance, Reporting and Programme Committee (FRPC) monitors the implementation of the management response.


8

Figure 1 outlines the policy cycle. Figure 2 outlines how evaluation can support policy activities.

Figure 1: The policy cycle

Consultation

Coordination

Decision

Implementation

Evaluation

Identifyingissues

Policy analysis

Policy instruments

The familiar diagram, from The Australian Policy Handbook by Caroline Althaus et al (2007), presents a picture of policy activity as an orderly progression arising from analysis and consultation, moving smoothly through decision into implementation and evaluation.

However, most policy and program workers experience a more unpredictable progression. Policy is often developed under pressure, sometimes in response to political events. Political considerations may impact on a policy’s progression to approval. Policy may be developed without the opportunity for consultation. Implementation may be rushed and there may be no opportunity for piloting.

Figure 2: Evaluation and policy

Demonstrate and provide evidence on the benefits to the policy• Public/stakeholder communication –

demonstrate value • Obtain further commitment and ownership

for intervention

Test and analyse current interventions and their performance• Formative evaluations • Annual and/or mid-term• Ex-post evaluation • Provide evidence of achievements to the

public and key stakeholders

Identify and analyse policy intervention options• What is the role of the Australian

Government/Department?• What do we already know (draw on existing

knowledge and expertise)?• What would success look like and how will

we get there? • Who else might need to be involved and how?• Analyse pros and cons for options as to best

solution to the problem• Evidence to support policy options

Identify and be clear on current/emerging problems• A structured way at looking at

problems/issues• Who and how will the problem impact?


9

The flowchart in Figure 3 outlines the Department’s evaluation (or regulatory review) process.

Figure 3: The Department’s evaluation process

How did the evaluation eventuate? Identified on the Departmental Evaluation Schedule

Line area drafts an Evaluation Plan

Line area provides an ED-cleared draft report and management response, to Governance Unit

Governance Unit arranges SC meeting, and prepares a cover paper, seeking consideration/endorsement of the report and the management response

Governance Unit provides the findings and approved management response, to Audit Committee members, for their information

Governance Unit uploads the SC and FRPC cleared report, management response and lessons learned, to the intranet/ENTR site

Governance Unit, if agreed by FRPC, uploads the report and approved management response on the Department’s website

Line area implements the recommendations in the approved management response

Line area conducts and manages the Evaluation: • Data collection • Regular progress reports • Draft report • Proposed management response • Lessons learned

Governance Unit arranges a Steering Committee meeting, and prepares a cover paper, seeking SC endorsement of the Executive Director-cleared Evaluation Plan

Governance Unit provides the report and management response to the Finance, Reporting and Programme Committee, for consideration/approval,

and, seeks agreement for public release

Governance unit provides a quarterly report to FRPC on the progress with implementation of the management response,

until full implementation/completion

Determined via consultation with Divisions and endorsed by executive

Line area may request use of a mentor (if/as required)

Governance Unit reviews/makes suggestions on the draft

Governance Unit manages the contract with the mentor and reviews

the progress reports

Governance Unit reviews/makes suggestions on the draft report, final report, draft management response

and draft lessons learned

Line area and mentor can approach the Governance Unit if any issues arise

throughout the Evaluation


10

Management responseThe management response is prepared by the line area that undertook the evaluation or regulatory review, and it is cleared by their Executive Director. It lists what it is that will be done, by whom, and by when, to address the recommendations or findings. The Department’s FRPC receives a quarterly update on the implementation progress of the management response, coordinated by the Governance Unit, with input from the relevant line area.

Lessons learnedThe ‘lessons learned’ is also prepared by the line area that undertook the evaluation or regulatory review. It highlights key points or issues arising from the findings and recommendations that staff can apply to their work; advice worth disseminating to other areas in the Department (in terms of how we design and implement policies/programmes/projects/regulations); and comments about the evaluation process itself (what went well and what was challenging).

TemplatesBasic templates1 are provided for:

• Logic diagram

• M&E framework and data matrix

• Evaluation plan

• Evaluation report

• Management response, and

• Lessons learned summary.

The intranet/ENTR site (under ‘evaluation’) includes electronic copies of these templates as a guide for staff to utilise and modify as required.

The detail included in templates will normally be covered and addressed by the in-house training provided by the Department.

Mentors can also provide guidance to line areas, for example, assistance in determining how to present information in a plan or report, or advice on how to select data sources and/or data collection methods.

Templates for regulatory reviews and best practice regulation can be found on the Office of Best Practice Regulation’s website (at dpmc.gov.au under ‘OBPR’).

1 Colour is used in these templates for presentation/training purposes only.


11

Logic diagram template

Inputs

Context and assumptions:

Activities/projects

The overall outcome:

Outcomes

Outputs


12

M&E framework and data matrix template

M&E framework template

1. Introduction and purpose of the M&E

Provide a brief description

2. Logic

Insert a copy of the policy/programme/regulatory logic diagram

3. Measures of success and data sources

List the success measures: which would indicate an outcome is being achieved.

List (ideally presented in a matrix) the data sources: where data and information will come from, how and when it will be collected, for example:

Source/Method A Source/Method B Source/Method C

Terms of Reference 1

Q1

Q2

Q3

Terms of Reference 2

Q1

Q2

Q3

4. Evaluation strategy

Outlines the approach to evaluation over the life of the policy/programme/regulation


13

Evaluation plan template

Evaluation plan template

1. Introduction and purpose of the evaluation plan

Provide a brief description about why the evaluation is being undertaken and what it will be used for

2. Terms of reference (TORs)

List the TORs

3. Key evaluation questions

List the key evaluation questions for what the evaluation will ‘ask’

4. Scope

Describes the focus of the evaluation, identifying what is in-scope as well as what is out-of-scope and the reason(s) ‘why’

5. Methodology

Indicates the methodology that will be used when undertaking the evaluation (referring to the M&E framework data sources, collection methods and matrix)

6. Timeframes and deliverables

Indicatesthekeystepsthatwillbeundertakenandwhen,includingwhenthedraftandfinalreportswillbe completed

7. Resourcing and management of the evaluation

Brieflydescribethecompositionoftheevaluationteamandrolesandresponsibilities;useofamentor; also providing a table of the budget that will be allocated for key tasks/phases such as data collection, consultation, analysis, reporting etc


14

Evaluation report template

Evaluation report template

1. Executive Summary

Provideaconciseoutlineofthefindingsandrecommendations

2. Introduction

Provide a brief description

3. Background

To the policy/programme/regulation

4. Methodology

Outline the key evaluation questions as well as the data and information that was collected and analysed (including both qualitative and quantitative means)

5. Key findings

Summarisethemainresultsandfindings

6. Recommendations

List the recommendations from the evaluation or review

7. Attachments


15

Management response template

Management response template

Recommendations or findings

Management response

(e.g. agreed/disagreed; and what actions will be

undertaken)

Responsibility

(e.g. Division/Branch)

Implementation timeframe

(e.g. by 30 June)

1.

2.

3.

4.

5.


16

Lessons learned template

Lessons learned template

1. Did the evaluation meet its purpose and were the TORs answered?

2. What main evaluation findings and recommendations are going to be relevant to the team or branch’s work in the future?

3. After doing the evaluation, what do you think would be useful for other areas of the Department to know about if they were to plan an evaluation?

4. Is there any other advice you would give, if the Department were to design and manage a new policy/programme/regulation in the future?


17

Examples of past evaluation-related activitiesThe following tables lists the evaluation-related activities undertaken in the Department over 2011–12 to 2015–16. The intranet/ENTR site (under ‘evaluation’) also provides electronic examples of these activities, and will be updated annually.

Evaluation activities undertaken from 2011–12 to 2015–16 (excludes training and workshops)

Priority Evaluations

Title Year Division

Heavy Vehicle Safety and Productivity Program (HVSPP) 2011–12 II

Regional Aviation Access Program (RAAP) 2011–12 AA

Keys 2 Drive 2012–13 STP

Strengthening Aviation Security Initiative (SASI) 2012–13 OTS

Roads to Recovery (R2R) 2013–14 II

Liveable Cities 2013–14 II

WA Service Delivery to Indian Ocean Territories (IOT) 2014–15 LGT

Asset Management 2015–16 Corporate & LGT

Indicative Evaluations

Title Year Division

Airport Tripartite Deeds 2014–15 AA

Regional Passenger Screening Programme (RPSP) 2014–15 OTS

Aviation and Maritime Security Identification Card Schemes (ASIC-MSIC) 2015–16 OTS

Last Ports of Call and International and Regional Capacity Building (LPOC) 2015–16 OTS

Review of the Departmental Evaluation Strategy 2015–16 Corporate

Regulatory Reviews

Title Year Division

Motor Vehicle Standards Act 1989 (MVSA) options discussion paper 2014–15 STP

Airport Curfew Administration Arrangements 2015–16 A&A

Continued ...


18

Evaluation activities undertaken from 2011–12 to 2015–16 (excludes training and workshops) continued ...

Logics and Monitoring and Evaluation Strategies (M&Es)

Title Year Division

National Stronger Regions Fund (NSRF) 2013–14 II

Bridges Renewal Program 2014–15 II

Community Development Grants 2014–15 II

Jervis Bay Territory Reform 2014–15 LGT

Regional Aviation Access Programme (RAAP) 2014–15 A&A

WA Service Delivery to Indian Ocean Territories 2014–15 LGT

Northern Australia Roads Programmes (Beef Roads and Nth Australia Roads) 2015–16 II

Drought Communities Programme 2015–16 II

Stronger Communities Programme 2015–16 II

Indian Ocean Territories (IOT) 2015–16 LGT

Norfolk Island Reform 2015–16 LGT

STP logics, M&Es and risk management; of each main activity 2015–16 STP


19

Reference listYear Title ISBN/Reference Author

2016 NSW Government, Program Evaluation Guidelines 978-0-7313-5499-3 NSW Government Premier & Cabinet

2015 Australian Government, Department of the Environment, Evaluation Policy 2015–2020

Creative Commons 4.0 Australia

Commonwealth of Australia

2015 Australian Government, Department of Industry and Science, Office of the Chief Economist, Evaluation Strategy 2015–2019

978-1-925092-67-7 Commonwealth of Australia

2015 Learning from Failure: why large government policy initiatives have gone so badly wrong in the past and how the chances of success in the future can be improved

978-0-9943750-0-1 Commonwealth of Australia,

Professor Peter Shergold AC

2015 Australian Government, Department of Finance, Resource Management Guide No. 131, Developing Good Performance Information

978-1-925205-17-6 Commonwealth of Australia

2014 Queensland Government, Queensland Government Program Evaluation Guidelines

Creative Commons

3.0 Australia

QLD Treasury and Trade and QLD Government Statistician’s Office

2013 Australian Government Department of the Prime Minister and Cabinet Cabinet Implementation Unit Toolkit 5: Monitoring, review and evaluation

- Australian Government Department of the Prime Minister and Cabinet, CIU

2013 Australasian Evaluation Society Evaluators’ Professional Learning Competency Framework

- Australasian Evaluation Society Professional Learning Committee

2013 United Nations Evaluation Group, Impact Evaluation in UN Agency Evaluation Systems: Guidance on Selection, Planning and Management

- United Nations Evaluation Group Impact Evaluation Task Force

2012 United Kingdom, Quality in qualitative evaluation: a framework for assessing research evidence (supplementary Magenta Book guidance)

978-1-84532-999-0 UK HM Treasury

2012 United Kingdom, Quality in policy impact evaluation: understanding the effects of policy from other influences(supplementaryMagentaBookguidance)

978-1-909096-49-3 UK HM Treasury; Department of Energy and Climate Change; and Department for Environment, Food and Rural Affairs

2011 United Kingdom, The Magenta Book, Guidance for Evaluation

978-1-84532-879-5 UK HM Treasury

Continued ...


20

Year Title ISBN/Reference Author

2011 United Kingdom, Major Projects Authority – AssuranceofbenefitsrealisationinMajorProjects– Supplementary guidance

UK Cabinet Office

2010 United Nations Evaluation Group Quality Checklist for Evaluation Terms of Reference and Inception Reports

- United Nations Evaluation Group

2010 United Nations Evaluation Group Quality Checklist for Evaluation Reports


2010 United Nations Evaluation Group Good Practice Guidelines for Follow-up to Evaluations


2010 The Canadian Evaluation Society, Competencies for Canadian Evaluation Practice

- Canadian Evaluation Society

2009 Australian Government, Department of Finance, Outcome Statements Policy and Approval Process

- Commonwealth of Australia

2005 United Nations Evaluation Group, Norms for Evaluation in the UN System


2003 United Kingdom, The Green Book, Appraisal and Evaluation in Central Government

- UK HM Treasury

Websites: www.betterevaluation.org, www.evaluationtoolbox.net.au, www.aes.asn.au

Reference list continued ...

www.infrastructure.gov.au INFR

A267

1

www.infrastructure.gov.au

Evaluation Resource Booklet - Department of Infrastructure ... · Creative Commons licence With the...

Documents

Transcript of Evaluation Resource Booklet - Department of Infrastructure ... · Creative Commons licence With the...