Horticulture Innovation Australia - Project M&E … · Web viewSection 1 Project M&E planning...

Project M&E planning guide and template Version 1 – September 2017

September 2017

Project M&E planning guideand template

A tool for delivery partners on how to undertake good project monitoring and evaluation


Content

Section 1 Project M&E planning guide 3

Purpose 4

Definitions 4

Monitoring and evaluation in the project cycle 4

Principles for project M&E 5

Planning M&E 6

What to include in your M&E plan 7

1 Program logic 7

2 Project M&E scope 12

3 Performance expectations 14

4 Data collection and analysis 14

5 Evaluation 16

6 Reporting and continuous improvement 16

Section 2 Project M&E planning template 17

Template structure 18

1 Program logic 19

2 Project M&E scope 19

3 Performance expectations, data collection and analysis 22

4 Evaluation 23

Reporting and continuous evaluation 24

Hort Innovation 1


Introduction The purpose of this document is to guide the development of project monitoring and evaluation plans by delivery partners, consistent with the monitoring and evaluation (M&E) expectations of Hort Innovation as outlined in Hort Innovation’s Organisational Evaluation Framework (2016)1.

This document includes two sections:

Project M&E planning guide: guidance on how to complete the template, including further information on Hort Innovation’s M&E planning expectations at the project level

Project M&E plan template.

© Horticulture Innovation Australia Limited 2017

Information contained in Project M&E planning guide and template has been developed by Hort Innovation in association with Clear Horizon Consulting and draws upon various sources as identified throughout the document. Project M&E planning guide and template should be used as a guide only.

Project M&E planning guide and template is published and distributed by Hort Innovation on the basis that Hort Innovation is not responsible for the results of any actions taken by users of information contained in this guide and template, nor for any error or omission from this guide and template. Hort Innovation expressly disclaims all and any liability and responsibility to any person, whether a reader or user of this guide and template or not, in respect of claims, losses or damages or any other matter, either direct or consequential arising out of or in relation to the use of and/or reliance, whether wholly or partially, upon any information contained in this guide and template.

The content of Project M&E planning guide and template is subject to change without notice. Any hard copies of Project M&E planning guide and template are uncontrolled documents and should not be relied upon as they may be out of date or inaccurate.

1 http://horticulture.com.au/wp-content/uploads/2016/12/Hort-Innovation-Evaluation-Framework.pdf

Hort Innovation 2

http://horticulture.com.au/wp-content/uploads/2016/12/Hort-Innovation-Evaluation-Framework.pdf


Section 1

ProjectM&E planning guide

Hort Innovation 3


PurposeProject M&E planning guide and template has been developed to support the implementation of Hort Innovation’s Organisational Evaluation Framework (Framework)2

at the ‘project’ investment level.

Its specific purpose is to guide delivery partners in developing an M&E plan, including a program logic model, for projects they are delivering. The relevant research and development (R&D)/Marketing Manager at Hort Innovation and the Data & Insights team will support project M&E planning by delivery partners.

DefinitionsMonitoring refers to the routine and systematic collection of data that may be used for management and/or evaluation purposes. Evaluation refers to the “systematic collection and analysis of data about processes, outputs and outcomes to allow us to make statements, judgments, claims and conclusions which have the potential to impact on current and future decision-making”3.

Monitoring and evaluation refers to the process of bringing together ongoing monitoring activities and intermittent evaluation studies into one overarching system. An M&E plan outlines the scope of M&E for a specific project, as well as what data will be collected (how, when and by whom) and how it will be analysed and used to meet identified M&E requirements.

Monitoring and evaluation in the project cycleMonitoring and evaluation are critical components of project management throughout the project cycle as shown in Figure 1. M&E enables accountability, demonstrates performance, facilitates continuous improvement and supports advocacy of R&D and marketing project outcomes (i.e. helps tell the project story).

2 http://horticulture.com.au/wp-content/uploads/2016/12/Hort-Innovation-Evaluation-Framework.pdf

3 Patton, M.Q. (1997) Utlization Focussed Evaluation, 3rd edition, Sage.

Hort Innovation 4


Figure 1: M&E in the project cycle

Principles for project M&EConsistent with Hort Innovation’s Organisational Evaluation Framework, project M&E should:

Be focused on the achievement of outcomes, i.e. on benefits to growers Articulate the logic of the project, clearly outlining how benefits will be delivered Identify performance measures (indicators) based on the logic of the project Be selective in what is measured to ensure that resources are used efficiently Be strategic, clearly linking to the Hort Innovation Strategic Investment Plan (SIP), relevant industry SIP

and Hort Innovation’s Organisational Evaluation Framework Be consistent where possible but sufficiently flexible to ensure that the specifics of each individual project

are considered Be commensurate with the size and scale of the project Be as practical, implementable and simple as possible Be used for continuous improvement.

Project M&E planning will be customised to the needs, size and requirements of each individual project.

Hort Innovation 5


Planning M&EHort Innovation projects are required to develop M&E plans. Exemptions may include short-term desktop studies, projects for which the purpose is evaluation of another project or program (impact assessment projects), where the project purpose is to provide data that feeds into a larger program, for example, the Minor Use Program, or where the project is delivering a component of a larger program, for example, digital advertising as part of a marketing program. M&E Plans developed at a program level will outline if and how contributing projects are responsible for collecting project-specific monitoring data.

In addition to accountability requirements, good M&E planning can help delivery partners adaptively manage their project to get the most out of the resources at their disposal.

Why plan? Monitoring takes time, effort and resources, so it is important that your effort (and those who you are working with) be used wisely. Carefully planning your monitoring ensures that you are using your time and resources efficiently and effectively. It also means that the data you collect is more likely to be useful to your project.

M&E plans will vary in length, complexity and approach depending on the project scope and value. The important thing is an M&E Plan is fit-for-purpose and meets the requirements of those funding and/or using M&E outputs.

Practical considerations to make at the planning stage include resourcing, timeframes, capability and capacity to undertake M&E, and the level of independence required of evaluation. These considerations will shape and inform what M&E is possible for a project.

Hort Innovation requires independent mid-term evaluations for projects that exceed an investment or risk level threshold4. The relevant R&D Manager or Marketing Manager will advise if an independent mid-term evaluation is required.

4 To be advised by Hort Innovation at procurement or during project delivery.

Hort Innovation 6


What to include in your M&E planThe project M&E plan template in Section 2 outlines what needs to be included in your plan. Hort Innovation M&E plans should include a consideration of the following:

1 Program logic clarifying the project impact pathway, and how the project is expected to contribute to Hort Innovation goals and the relevant industry SIP outcomes

2 Scope, including purpose and audience for the M&E, outlining who will use the M&E results and what they will use it for. M&E Plans often use key evaluation questions (KEQs) to guide data collection

3 Performance expectations for assessing and judging project performance against agreed indicators or measures

4 Monitoring data collection and analysis arrangements including methods (and specific data sources and responsibility for collection)

5 Evaluation data collection and analysis arrangements including methods (and specific data sources and responsibility for collection)

6 Data collation and synthesis processes, bringing the information from monitoring and evaluation together to make an evaluative judgement

7 Reporting arrangements.

Each of these requirements is outlined in more detail in the following sub-sections. The project M&E plan template in Section 2 provides cross-references to these sub-sections to support completion of the template.

1 Program logic

What is program logic?

Program logic is a thinking tool used by Hort Innovation to:

Consider how outcomes can be achieved through investment (the causal chain or impact pathway) Guide and inform investment selection Monitor, evaluate and report on progress and achievements of investment.

Figure 2 shows a simplified program logic model. The parts in green are the essence of the logic of your project.

Figure 2: Simplified program logic model

Hort Innovation 7


Table 1 outlines the Hort Innovation levels and definitions for program logic at the ‘project’ investment level.

Table 1: Project level logic hierarchy

Strategic Investment Plan outcome(s)

SIP outcome(s) that the project is ultimately expected to contribute towards. These are articulated in the SIP relevant to your project. The project must demonstrate contribution to at least one outcome of the SIP the project relates to. SIP outcomes are economic, environmental or social impacts. Example: Increased consumer demand through improved product quality.

The ‘line of accountability’

End-of-project outcome(s)

These outcomes are the desired result of the project, and represent the project’s unique contribution to the relevant SIP outcome(s). They represent what the project could be reasonably expected to achieve given the level of investment and the timeframe for delivery (though some outcomes may be realised after the project period). Includes: practice or behavioural changes; adoption; incremental improvements stimulated through R&D; use of new information/protocol/technology; increased recognition of Australian horticulture products; increased consumption by a specific demographic; improved product quality; commercialised intellectual property (IP). Example: An increase in product meeting minimum quality standards.

Intermediate outcomes

Short to medium term outcomes brought about through project activities, and which describe the cause-and-effect pathway through which the activities support the achievement of the end-of-project outcome(s). Includes: changes in grower knowledge, attitudes, skills & aspirations (KASA); marketing campaign reach and influence; access to new information; training accessed by growers. Example: Industry stakeholders aware of quality standards and initiated implementation within their business.

Activities and outputs

The activities delivered by the project to bring about the desired changes, and the immediate effects – or tangible deliverables/outputs (products or services) – of those activities. The activities are what projects are funded to do. Will include: publications; data; field trial and experiment results; minor use permits; new farm management protocols or standards; new technology; marketing campaigns and collateral; training/extension events and number of growers trained/attended; industry development services; trade shows; study tours; reports; patents; prototypes; new varieties or technology. Example: Publication of quality standards, training of growers, and quality standards testing.

Foundational activities

Preliminary or ‘preparatory’ activities that are an important for the success of the project but are not directly associated with changing or influencing the external environment as project activities are. It includes establishing the corporate operational and business processes to support R&D, marketing and industry development, i.e. industry SIP and development of an industry marketing plan. It also includes: establishing a project team; outputs from previous or linked projects; data sets; a new variety, protocol or technology that is to be extended in your project; scoping study results.Example: Development of a quality standard, formation of project team and establishment of quality standard benchmarks.

Hort Innovation 8


Developing your program logic model

Figure 3 provides a template for the completion of the logic model for your project. The development of a logic model involves the following steps, ideally in the following order:

1 Clarify the higher-level outcomes for your project The higher-level outcomes are divided into two levels, above and below the ‘line of accountability’ (see

Table 1 above): The broader goal(s), i.e. SIP outcome(s) that the project is expecting to contribute towards, sit above

the ‘line’. They represent the SIP outcome(s) most relevant to your project The end-of-project outcome(s) are just below the ‘line’, and represent what your project is directly

responsible for achieving. They show what success would look like for your project, if it achieved what it was expected to achieve, and represent the unique contribution of your project to the broader goal(s)

Note, there is no ‘right’ number of broader goal(s) or end-of-project outcome(s) – the test is that they capture what your project intends to achieve and what it intends to contribute towards.

2 Identify intermediate outcomes

Identify the pre-cursors or stepping stones that, if achieved, would lead to the end-of-project outcomes. Check them using ‘if-then’ logic, i.e. if quality standards are published, then industry stakeholders will be aware of the standards.

3 Determine activities to achieve outcomes

Distinguish between activities that influence change from those that are related to planning and preparatory work and which are foundational to your project, but which don’t create change in and of themselves

Ideally, if being strategic, we work backwards from the broader goals to identify what activities would be appropriate to achieve the outcomes we want to. Most organisations have a clear idea of the activities they want to undertake. In this case, the activities are listed out before the intermediate outcomes and then the intermediate outcomes are identified.

4 Check assumptions underpinning the theory of your project

It’s very important to step back and check what assumptions are being made about how your project activities will lead to the outcomes you expect to achieve

Document the key assumptions, for example, likely adoption rates5 underpinning your project as well as:o The evidence you have for the assumptiono Your confidence that the assumption will ‘hold true’o The risk to achieving the end-of-project outcomes if the assumption turns out to be wrong

Attention needs to be paid to assumptions for which you have low confidence and are also a high risk for achieving the end-of-project outcomes should they turn out to be wrong. This normally means the project is unlikely to achieve its outcomes and the design of the project should be reconsidered.

5 The CSIRO developed ADOPT (Adoption and Diffusion Outcome Prediction Tool can assist those involved with agricultural research,development and extension to apply and understand factors that are likely to affect adoptability(https://research.csiro.au/software/adopt/)

Hort Innovation 9


Note: Be careful not to simply ‘fill in the boxes’ in the template. Program logic is a thinking tool that results in a documented model, but the template for the model does not do the thinking for you. The template is to document your thinking. Not all projects will operate at all levels of program logic. For example, some projects will provide an output, such as a dataset, that becomes a foundational input to another project that seeks to achieve practice change. Some projects provide access to new information or technology but do not seek to influence change or adoption.

Hort Innovation 10


Figure 3: Program logic model template

Hort Innovation

SIP outcome(s) that your project is ultimately expected to contribute towards Relevant SIP outcome(s)

FundingProjectplanning

Establishing partnerships

Baseline data collection

Project administration

E.g. a new variety, new protocols, data, etc from a related or preceding project

What you do; i.e. work performed including

experiments, field trials, extension events,

training, technology development, etc







Tangible deliverables as a direct result of your

activities; i.e. publications, marketing collateral, growers

trained



trained



trained

Short- to medium-term outcomes brought about through project activities, and which describe the cause-and-effect

pathway through which the activities support the achievement of the

end-of-project outcome(s)

Short- to medium-term outcomes brought about through project activities, and which describe the cause-and-effect

pathway through which the activities support the achievement of the

end-of-project outcome(s)

The desired final result of your project – and which represents your project’s unique contribution to the relevant SIP outcome(s). Sometimes there is more than one end-of-project outcome


Foundational outputs

Activities

Outputs


End-of-project outcomes

11


2 Project M&E scope

Scope includes the M&E audience and their information needs; the key evaluation questions that will guide data collection for the project; and the M&E budget (resources available for M&E).

2.1 Audience

Here, we identify who wants to know what about the project, and distinguish between primary and secondary audiences:

Who is the primary M&E audience for this project (those who will use the results of M&E to manage and/or make decisions about the project, for example, project team, Hort Innovation R&D Manager) and what do they want to know about the project?

Are there any other (secondary) M&E audiences for this project (those who may be interested to know the M&E results of the project, but do not have a management or decision-making role in relation to the project)?

The primary audience always includes the project team as the team needs monitoring information to ensure the project is on track and make any adjustments along the way. It also always includes Hort Innovation, as the funder of the project.

The secondary audience of M&E for Hort Innovation projects include co-investors, project partners and project reference groups.

Project-level M&E information is used by Hort Innovation to do its own reporting to levy payers and other stakeholders.

The information needs identified for each audience should be reflected in the monitoring plan.

2.2 Key evaluation questions

KEQs are overarching questions that guide evaluative inquiry into the effectiveness, efficiency, appropriateness, relevance, impact and legacy of Hort Innovation’s investments. They facilitate judgements and are used to guide an evaluation. They form the basis of data collection but are not the same as survey questions.

Table 2 outlines the KEQs that have been determined in Hort Innovation’s Organisational Evaluation Framework as most relevant to guide project level evaluation.

Those KEQs most relevant for your project will depend on who you have identified as the primary M&E audience for your project and their information needs.

The KEQs should be tailored to each specific project, as shown in the project M&E plan template in Section 2.

Hort Innovation 12


Table 2: Hort Innovation’s KEQs relevant at the project level

Domain* Key evaluation questions

Effectiveness 1 To what extent has the project achieved its expected outcomes?

Relevance 2 How relevant was the project to the needs of intended beneficiaries?

Processappropriateness

3 How well have intended beneficiaries been engaged in the project?4 To what extent were engagement processes appropriate to the target audience/s of the project?

Efficiency 5 What efforts did the project make to improve efficiency?

* Definitions of the ‘domains’ are listed below

Definitions of domains

Effectiveness: the extent to which an intervention (a project) has attained (or is expected to attain) its intended outcomes.

Relevance: the extent to which the expected outcomes of an intervention are consistent with beneficiaries’ requirements, government priorities, etc.

Appropriateness (process): the extent to which a project/program is operating as intended.

Efficiency: the extent to which an intervention produces outputs and outcomes without wasting time, money, effort or other resources.

2.3 M&E budget

Depending on project arrangements, a delivery partner in most instances will be responsible for implementing the M&E for their project. A delivery partner (or Hort Innovation) may in some instances contract or sub-contract parts of their project M&E out to a third party. Hort Innovation may contract out responsibility for an entire evaluation to a third party (for example, development of an M&E plan for a program/group of projects, evaluation of a cluster of projects, mid-term evaluations).

In most instances, Hort Innovation expects that delivery partners will conduct M&E internally with outside support and advice as required. Who implements M&E will ultimately be determined at the planning stage where practical considerations are often very important. At a minimum, delivery partners are expected to collect evidence (monitoring) that can be used to make a self-evaluative judgement at the project conclusion and as an input to Hort Innovation’s evaluations.

Hort Innovation 13


3 Performance expectations

While program logic identifies how change is expected to occur because of investment, it does not stipulate how much change will be necessary and sufficient to deliver expected benefits to growers and levy payers. Performance expectations such as indicators, standards, or measures can be set for assessing the performance of the project in achieving its intended outcomes. Performance expectations can be expressed quantitatively as a measure or an indicator, qualitatively as a set of standards or criteria, or as a mix of quantitative and qualitative descriptions of success. In addition, where appropriate, outcomes can be re-written to include a target, for example, 20 per cent of growers adopt (insert specific) practice by (insert timeframe).

Performance expectations for assessing project effectiveness suggested in Hort Innovation’s Organisational Evaluation Framework are outlined below. These indicators and measures are not exhaustive and should be customised depending on the needs of each project.

X per cent of growth in retail value of product by [date] X per cent of growth in retail volume of product by [date] Increased yield by X per cent (by industry segment) X per cent in increased sales X per cent of audience reached (by market segment) Increased consumer awareness and brand recognition Changes in consumer perceptions of quality Extension participation (X per cent of growers involved) Practice change by X per cent of growers (by industry segment) Influence of research uptake on growers and advisers (by industry segment) X per cent increase in value of horticulture exports by [date] Completion and availability of a new variety/protocol/technology Data reports provided monthly X editions of industry publication per annum.

4 Data collection and analysis

Once the scope of project M&E, a logic model (or equivalent) and performance expectations have been defined, it is necessary to set out how data and evidence will be collected for M&E purposes. Data collection may involve a range of quantitative (numeric) or qualitative (descriptive, text-based or visual) information. The important thing is that data collection methods are fit-for-purpose; i.e. they address the information needs as defined by the KEQs and performance expectations. To address gaps or deficiencies in data quality it will often be useful to use multiple data collection methods.

Note that while it may be desirable to set a baseline at the beginning of the project or use benchmarks to monitor progress against, it is not always possible to do so due to insufficient data. Moreover, broader industry trends are often influenced by a range of factors outside of the control of projects. To address these complexities, technical evaluation approaches such as correlation analysis and/or contribution analysis may be used.

Hort Innovation 14


Hort Innovation 15


A suggested format for identifying data collection methods and sources is provided in Table 3. It is provided as a suggested format for linking project outcomes, performance measures and data collection methods. This table may be customised and modified as necessary for each project.

This table provides the core information for a well thought out and effective project M&E plan. To ensure that the M&E system is balanced and appropriate it is necessary to identify what to monitor at all levels in this plan; such as, end-of-project and intermediate outcomes, as well as outputs delivered by the project. In addition, it is critical that data collection methods are fit-for-purpose (directly linked to performance measures and outcomes). To this end, it may be useful to include multiple methods including qualitative and quantitative approaches. For further guidance, arrange with your R&D or Marketing Manager to discuss directly with the Data and Insights team.

Table 3: Project monitoring plan

Program logic level

What to monitor Insert aspects of the project to be monitored, for example, grower groups, trials, improved awareness or knowledge, application of a new technology

Performance expectations KPIs

How to monitor Suggestedmethods

Data sourceTarget audience

When Responsibility Who is responsible for the monitoring and how will results be reported

ActivitiesWhat you do, such as field trials, farm days

# trials # extension activities

Record keeping

Growers Advisors

Ongoing Organisation/specific project team member

Milestone Reports

Final Reports

OutputsTangible deliverables as a direct result of your activities

# articles # publications Extension

participation (# growers)

# minor use permits New technology or

variety or protocol

Record keeping

Growers Advisors Exporters Packers

Ongoing Organisation/specific project team member

Milestone Reports

Final Reports Industry reports/

publications

Intermediate outcomesChanges in knowledge, attitudes, skills and aspirations (KASA)

Increased knowledge/use of information by # growers

#/% audience reached

Event questionnaire

Observation Interviews Surveys Case studies Website/

marketing analytics

Growers Advisors Processors

Intermittent;i.e. annually

As required for evaluation

Organisation/specific team member

Independent reviewer

Milestone Reports

Final Reports Industry reports/

publications

End-of-projectoutcomes Levy and co-investment funded

research, development, extension and marketing and extension outcomes

Uptake and adoption targets because of the project

# practice changes by x% production base

Increased yield by x%

Increased consumer recognition

x% increased sales/exports

x% increase in water use efficiency

x% increase in product meeting minimum quality standards

Interviews Survey Case studies Industry data/ benchmarks

Growers Exporters

As required for evaluation

Organisation/specific project team member

Independent reviewer

Final Report Evaluation

Report

Hort Innovation 16


Data collection methods include:

Benefit cost analysis Project recordsSurveys InterviewsBenchmarking ObservationPublication records Website analyticsParticipant feedback sheets Case studies/narrativesExtension event tools such as dartboards Feedback from project reference groups

5 Evaluation

Evaluation comes in two forms. It may involve:

Making evaluative judgements using the monitoring data collected and any additional data collected at the point of evaluation, and answering your key evaluation questions through self-assessment or expert judgement

Commissioning an external evaluation study to review existing data, collect additional data and make evaluative judgements.

Both types involve making judgements and this is what defines evaluation.

Evaluation refers to a process that involves making judgments about the worth or merit of aspects of a project or program. This may be focused on several different evaluation questions. One way of thinking about evaluation is that it is about answering the ‘so what’ questions.

At a minimum Delivery Partners are expected to self-assess and make evaluative judgements of their project (i.e. answer the KEQs) at project conclusion using the M&E data collected. Hort Innovation will require independent mid-term and final evaluations for some projects.

6 Reporting and continuous improvement

Reporting

It will be necessary for each Hort Innovation funded project/program to communicate their performance to Hort Innovation as well as advocate the results of their projects to other interested parties, e.g. industry stakeholders (i.e. growers/levy payers). The results of M&E underpin this reporting. Reporting is required through a range of reporting products, including Milestone Reports, Final Reports, articles in industry magazines, conference presentations, Hort Innovation publications and website. Hort Innovation will use reports provided by Delivery Partners in the preparation of the Hort Innovation Annual Report and to inform ex-post impact assessments.

Continuous improvement

To ‘close the loop’ it is good practice in M&E planning to outline how M&E information will be used for improvement purposes, for both the project and the project M&E plan itself. This can be done by simply describing how M&E results will be used for communication (i.e. to growers), reflection and improvement purposes. This is just part of good project management.

Hort Innovation 17


Section 2

ProjectM&E plan template

Hort Innovation 18


Template structureThe template consists of several sub-sections that collectively form a comprehensive Project M&E Plan:

1 Program logic2 Scope of project M&E3 Performance expectations, and data collection and analysis4 Evaluation5 Reporting, learning and improvement.

Note: Each sub-section is cross-referenced to the corresponding ‘how to’ instructions in Section 1: Project M&E Planning Guide (the guide). Please read these instructions to support your completion of the template.

As you complete the template to create an M&E Plan for your project please delete all guidance and help text.

Please contact the R&D or Marketing Manager for your project for a Word version of the template.

Hort Innovation 19


1 Program logic For assistance, refer to Section 1: Project M&E Guide – part 1.

Figure 1 Logic model for project

Insert your program logic here, using the template provided in Figure 3 of the guide, showing:

the ‘end-of-project’ outcomes your project expects to achieve (‘below the line’) the Strategic Investment Plan (SIP)6 outcome(s) your project expects to contribute towards

(‘above the line’) the activities and outputs the project will deliver how those activities and outputs are expected to cause the end-of-project outcomes (intermediate

outcomes).

2 Project M&E scopeFor assistance, refer to Section 1: Project M&E Guide – part 2.

2.1 Audience

For assistance, refer to Section 1: Project M&E Guide – part 2.1.

Complete the M&E audience and their information needs for your project in Table 1.

Table 1: M&E audience and their information needs

Audience Information need

Primary

Project team

Hort Innovation Examples only: Project progress such as number of trials completed; Extent of levy payer engagement in the project; Extent of adoption.

Etc.

Secondary

e.g. Co-investors

e.g. Project partners Examples only: Number of experiments completed; Forecast of team member allocation to tasks; Project expenditure.

6 Strategic Investment Plans (SIPs) are available at http://horticulture.com.au/grower-focus/

Hort Innovation 20


2.2 Key evaluation questions


Tick those KEQs that are relevant to your project in Table 2, and, where appropriate, write the selected questions out with more specificity for your project.

Note: The R&D or Marketing Manager for your project can help you select the KEQs relevant to your project.

Table 2: Project key evaluation questions

Key evaluation questions Relevant? Project-specific questions

Effectiveness

1. To what extent has the project achieved its expected outcomes?

Examples only (note that while several examples have been provided there is usually only one overarching effectiveness question):

To what extent has the project increased the adoption of the industry Best Practice Guidelines?

To what extent has the project improved knowledge and awareness of consumer insights?

To what extent has the project increased the % of produce meeting minimum quality requirements?

Has the project developed new technology that is now available for industry uptake?

Relevance

2. How relevant was the project to the needs of intended beneficiaries?

Example only:

To what extent has the project met the needs of industry levy payers?

Process appropriateness

3. How well have intended beneficiaries been engaged in the project?

Examples only:

To what extent were the target engagement levels of industry levy payers achieved?

Have regular project updates been provided through linkage with the industry communication project?

4. To what extent were engagement processes appropriate to the target audience/s of the project?

Examples only:

Did the project engage with industry levy payers through their preferred learning style?

How accessible were extension events to industry levy

Hort Innovation 21


Key evaluation questions Relevant? Project-specific questions

payers?

Efficiency

5. What efforts did the project make to improve efficiency?

What efforts did the project make to improve efficiency?

Other (if any)

Include any project-specific question(s) determined important for your project here

These may relate to project impact or legacy.

2.3 M&E budget


List the financial and human resources available for implementation of the M&E plan, i.e. for monitoring data collection, analysis, synthesis and reporting, as well as any additional data collection (and analysis, synthesis) that might be required at the point of evaluation. It is expected that delivery partners in most instances will be collecting monitoring data through normal project activities and record keeping.

Hort Innovation 22


3 Performance expectations, data collection and analysis For assistance, refer to Section 1: Project M&E Guide – parts 3 and 4.

Describe any performance expectations against the activities, outputs and outcomes of your project in Table 3, as well as the data collection details (method, source, timing and responsibility). This should provide the bulk (if not all) of the monitoring information you need to understand and demonstrate project effectiveness.

Note: Not all activities, outputs and outcomes need to be monitored - be selective. Refer to what the primary audience wants to know.

Table 3: Project monitoring plan

Logic level What to monitor Performance expectation (KPIs) and/or monitoring questions

Data collection – method (e.g. survey) and source (e.g. growers)

Timing of, and responsibility for, data collection


e.g. Provision of data or development of a new management protocol in a related project; Formation of project team

e.g. Monthly provision of data

e.g. Record keeping (Delivery Partner of related project)

e.g. Monthly (Project Leader)

Activities and outputs

e.g. Field trials; Technology development; International engagement; Workshops; Publications; Training packages; Number of growers accessed training/attendance at extension events; Industry development services; Minor use permits; New varieties; New standards or protocols

e.g. Extension participation (number of growers and other stakeholders)

e.g. Record keeping (Industry Development Officer)

e.g. Ongoing (Industry Development Officer)


e.g. Changes in knowledge, attitudes and skills of growers on a specific best practice; Access to new information for business decision making

e.g. Number of growers indicating an increase in knowledge of how and why a best practice should be implemented

e.g. Event questionnaire (Growers)

e.g. Intermittent and as required for Milestone Reports (Project Team member)

End-of-project outcomes

e.g. Uptake and adoption of a specific best practice by growers; Implementation of a new protocol; A change in value/volume/quality

e.g. Practice change by a target percentage of production base (and result of that

e.g. Surveys and case studies (Growers)

e.g. Annually and at end of project for Final Report (Project

Hort Innovation 23


practice change) Leader)

4 Evaluation For assistance, refer to Section 1: Project M&E Guide – part 5.

Depending on the size of your project and therefore its evaluation requirements, describe here either any additional data that you will collect to answer the KEQs at project completion (self-evaluation) (Table 4), and/or what independent external evaluation study(ies) will be required (Table 5). Hort Innovation will advise if an independent external evaluation is required.

Note: A well-designed monitoring plan usually provides the information you need to demonstrate project effectiveness. Any data collected at the point of evaluation usually complements the monitoring data to either answer the effectiveness KEQ more fully and/or answer other evaluation questions identified in Table 2.

Table 4: Additional evaluation data requirements

KEQ Data collection requirement Source and method

Copy from Table 2 If no additional requirement, state ‘NA’

Copy from Table 2 If additional requirement, state here along with source and method.

Examples: Number of new export markets accessed; Final adoption figures of a new best practice with associated cost benefit data; Increase in product achieving a minimum quality standard from project initiation to completion

Table 5: Independent evaluation studies (as required by Hort Innovation)

Type of evaluation When (start and finish)

Mid-term evaluation Insert timing

Final evaluation Insert timing

Hort Innovation 24


5 Reporting and continuous improvement For assistance, refer to Section 1: Project M&E Guide – part 6.

List the report(s) will you prepare, to whom and when.

Note: Apart from usual project reporting, consider any reporting or communications to secondary audiences.

Table 6: Project progress reporting

Report type To whom Timing

e.g. Milestone Reports Hort Innovation Six-monthly

e.g. Final Reports Hort Innovation At end of project

e.g. Articles Industry magazine Annually

e.g. Written and verbal update Project Reference Group Six-monthly

e.g. Financial reports Project Partners Annually

What learning and improvement process(es) will your project use?

Table 7: Project continuous improvement activities

Continuous improvement process

Details Timing

e.g. Reflection meeting with Hort Innovation R&D Manager

Meeting between R&D Manager/Marketing Manager and Delivery Partner to discuss progress to-date and what’s working well/not, and agree any follow up actions

Six-monthly

e.g. Team meetings Meeting between project team members to discuss project trials and their timing/Meeting between project team members to discuss feedback from extension event participants to determine gaps in adoption and preferred learning styles for incorporation into project

Quarterly

e.g. Project Reference Group meetings

Meetings between project team members, Hort Innovation and industry representatives to gain feedback on project activities and refine methodology

Annually

Hort Innovation 25

Horticulture Innovation Australia - Project M&E … · Web viewSection 1 Project M&E planning...

Documents

Transcript of Horticulture Innovation Australia - Project M&E … · Web viewSection 1 Project M&E planning...