Introduction to monitoring and evaluation Ungaluk Program 2015

For Ungaluk Funded Projects

Monitoring =

Keeping track of the implementation and performance of a project.

It requires collecting information on the project activities. E.g. collecting information on performance of participants

in a project throughout the project.

It is important to track if the project is meeting its objectives, and if any changes are needed.

The information collected during the monitoring phase is used for the final evaluation of a project.

Evaluation =

It is where you:

monitor what you are doing

measure what you have done

find out what was effective and what was not

Uses information collected during monitoring (during the project) AND/OR after the project

Can answer questions that need to be answered for good project management.

For example, you might ask: Are we reaching who we wanted to reach or are

we missing the people who most need our project? Do we need more project staff? Do project staff need more training? Is the project resulting in the changes we thought

it would? What aspects of the project should be changed,

expanded, continued, or discontinued?

There are three phases in a project lifecycle:

Phase 1 - Needs assessment

Phase 2 - Project planning

Phase 3 – Implementation

You can do an evaluation at any phase. For Ungaluk, most projects will be evaluated at Phase 3 since they are being funded to be implemented.

Monitoring and evaluation are important for identifying:

Challenges during the project

Impacts and results of the activities

What works and what does not work

Lessons learned and how to improve the project

“Doing” crime prevention is an ongoing process that requires a long-term vision so that there is a long-term impact. We can only know if we have an impact by doing M&E.

If we care about preventing crime and victimization in our communities, it only makes sense to care about what works in reducing crime and victimization. The only way to know for sure what works and what doesn’t work is to invest in evaluation.

Three general types for implemented projects (phase 3):

Process evaluation

Outcome evaluation

Impact evaluation

Looks at the development and implementation of a project.

It shows if you’ve hit measurable targets and implemented strategies as planned.

It’s typically done at the end of the project and it looks at the project from start to finish, assessing cause-and-effect relationships between the project components and outcomes.

This type of evaluation can be very useful in determining whether a project should be continued, expanded upon, or eliminated.

For example, your process evaluation might confirm that 200 people have completed your skills-training program.

Outcome evaluation measures the change that has occurred as a result of a project.

For example, an outcome evaluation would tell you how many of the 200 people that completed your skills-training program actually showed an increased confidence, changed behaviours, found jobs because of the new skills, etc.

Looks at the long-term, deeper changes that have resulted from that project.

This type of evaluation could suggest that there were changes to the lives of the participants that did the skills-training, and this change continued over time and maybe transferred across generations.

1. Identify goals (anticipated outcomes)

2. Describe the project (list project activities)

3. Identify what you want to know (evaluation questions) Determine what information you will need to collect:

To see how your project is doing day to day (on-going monitoring)

To see if you are on track to achieve your results, if you are on time and if you are using resources as planned mid-way through your project (mid-term evaluation), so that you may make changes as needed

To see if the overall changes you were trying to achieve actually happened by the end of the project (final evaluation) and identify what you learned.

4. Identify data sources and data collection tools (See **These are some examples of sources of information – how to collect the information)

5. Collect the information

6. Organize the information

7. Analyze the data

8. Report the results, identify next steps

II. Plan You will be provided with monitoring and evaluation tools, and examples of sources of information on how to collect the information. The evaluation plan includes the information that will

be collected throughout the project and the indicators used to evaluate this information. The project manager will complete a final evaluation report by analyzing the information provided in the plan to present the results of the project.

Please fill out the following information detailing your monitoring and evaluation plan which will be used for the final report. Please use some of the indicators below.

Project managers can use their ‘Work Plan’ in the application form (#10 + #11 in the application form) to fill out Column 1 : “What will be monitored and evaluated? “

Column 1

What will be monitored and evaluated? [What do you want to know about your project?]

List the objectives and specific activities

Column 2

How will it be monitored and evaluated? [How will you know that the project is achieving its objectives and outcomes?]

See the list of example indicators below *

GO TO TABLE p.5-7:

*These are some examples of indicators for all projects and for specific priority areas and objectives:

Column 3

How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used 3. How often the information will be collected

GO TO LIST p.8: **These are some examples of sources of information – how to collect the information:

Column 4

What are the expected results? [What will happen as a result of your project?]

When do you expect to see the results?

What will be monitored and evaluated? [What do you want

to know about your

project?]


How will it be monitored and evaluated? [How will you know that the project is achieving its

objectives and

outcomes?]


How will the information be collected? List the: 1. Source of information (See below for examples**) 2. Tools used

3. How often the

information will be

collected


When do you expect to

see the results?

Objective 1: To reduce the incidence of dating violence among high school youth.

Objective 2: To increase public awareness and understanding of dating violence as a serious issue.

Indicators:

Number of stakeholders

who attend planning

meetings

Number of

volunteer/in-kind

hours spent on

collaborative efforts

1. Project records

2. Minutes taken at

meetings AND

Time sheets

3. Monthly and

Ongoing

Results:

Increased involvement

of community

stakeholders in

collaborative efforts to

reduce dating violence

When get results:

End of the project

1. We need to know the project's goals, priority area and clientele it is trying to reach.

2. We need to develop a work plan, that shows the relationships between the project’s activities, outputs or results :

Planned Activities

E.g. Recruiting participants, hiring staff, organizing workshops, providing services etc.

Tasks

How, by whom, with what etc.

TimelineExpected results or outputs

E.g. Services or products developed by the activities, outcome for the participants etc.

Identify your project goal and who your clientele. Identify the objectives that will lead to your goal. Determine what the components of your project will be -

that is, your broad strategies or service areas. Describe the project inputs. Who and what will be required

to operate your project? For each component, describe your activities. Who will do

what, and when? Identify the outputs of your activities. How many

participants do you expect? What (and sometimes, how many) tools, materials, or events will be produced?

Identify the outcomes/results linked to these activities. Remember that outcomes represent changes you hope to see result from your activities; they are not just the delivery of the activities themselves. You will want to mention the short-term and intermediate

outcomes of your activities, making sure that these in turn link to your overall goal(s).

3. Define the purpose of your evaluation and the questions you want it to answer

4. Define indicators that will show your project is achieving its expected results/outcomes

5. Identify sources of information for these indicators

6. Determine how you will gather the information

3. Identifying evaluation questions

Determine the goal of the evaluation (not of the project) – This will give you an idea of the questions you will want the evaluation to answer.

Find out what the funder, staff, participants, partners, and others want to know.

Here are some ideas: Was the project implemented as planned?

Did the priority clientele access the project?

Did the project achieve its purpose (expected results)?

Were there unexpected results of the project (positive or negative)?

4. Identifying indicators

What is an "indicator"?

A variable (or information) that measures one aspect of a project.

It indicates whether a project has met a particular goal/objective.

There should be at least 1 indicator for each outcome/expected result.

There are two kinds of indicators:

A process indicator provides evidence that a project activity has taken place as planned.

An outcome indicator provides evidence that a project activity has caused a change or difference in behaviour, attitude, etc.

So, an indicator must be something we expect to change from the time the project begins (known as the baseline) until a later point when the project activities have taken place and are likely to have had an impact.

Indicators should:

provide accurate and reliable evidence

be easy to gather

provide useful information for making management decisions


to know about your

project?]



objectives and

outcomes?]



3. How often the

information will be

collected



see the results?

Objective 1: To reduce the incidence of dating violence among high school youth.


Number of stakeholders

who have "signed on" to

the project

1. Letters of

commitment

AND

Project Plan

AND

Literature

2. Comparison of

project plan to best

practices identified in

the literature

3. End of project

Results:

Enhanced commitment of

key stakeholders to a

comprehensive, evidence-

based plan for addressing

dating violence in the

community

When get results:

End of the project


to know about your

project?]



objectives and

outcomes?]



3. How often the

information will be

collected



see the results?


Level of knowledge

about factors leading to

dating violence among

youth

1. Participants

2. Test of participant

awareness of the

factors leading to

dating violence

3. Before project

sessions start and

again when they are

finished

Results:

Increased participant

awareness about factors

that contribute to teen

dating violence

When get results:

Mid-term of the project

5. Identifying information sources

Once you have identified your indicators, you will need to think about who will provide the information you need. It's best to use a number of sources of information.

Some examples are: Participants – intake form, interviews, focus groups,

observation Public – surveys, questionnaires, community-level

statistics Other agencies – focus groups, key informant interviews,

surveys Project staff – focus groups, key informant interviews,

project records/notes Media – review of media reports

6. Choosing data collection methods

How can you get the information?

Project records/document review

Interviews/focus groups

Surveys/questionnaires

Participant observation

Population level data/statistics

Qualitative data:

It is in the form of words, but may also include any information that is not numerical in form, such as photographs, videos and sound recordings.

Describes a specific context, event, people or relationship in a broad way, by trying to understand the underlying reasons for behaviour, thoughts and feelings.

Examples: observation, one-on-one interviews, focus groups and case studies.

Quantitative data

It is in the form of numbers.

Used to count events or the number of people who represent a particular background.

Examples: surveys, questionnaires and statistical data (such as census information).

In all cases, it's important to have informed consent for any information collected.

If information collection includes photographs or videos of participants, always obtain participants' permission to use the photos/videos in whatever way is anticipated.

When collecting information from or taking pictures of children and youth (under 18 years of age), first obtain permission from their parents.

Some options include:

Continuously

After each event/activity

At regular intervals

Before and after projects

You should try to gather information:

before the project/activity begins (or soon after it begins) and

after it is complete

http://www.makivik.org/monitoring-evaluation-tools/

Select:

“Evaluating Crime Prevention through Social Development Projects: Handbook for Community Groups”

http://www.makivik.org/monitoring-evaluation-tools/

1. Evaluation Planning Tools

2. Evaluating Crime Prevention through Social Development Projects: Handbook for Community Groups

3. Australian Government – Evaluating community crime prevention projects

4. Australian Government – Monitoring and evaluating your project tip sheet

In this resource, the following is provided:

Information on planning a project and the project phases

Information on developing an evaluation plan

Examples of a project plan and an evaluation plan


More details than the previous resource on - Planning a project and the project phases Developing a logic model as a project plan Developing an evaluation plan

It also provides information on - Data collection methods Analyzing data/information How to report the data/information Challenges and solutions to evaluation Other sources of information


Step-by-step guide to making an evaluation plan and evaluating a project

Developing an action plan on what to do with the findings from the evaluation

Managing the evaluation


A few basic tips on monitoring and evaluating a project

A few basic questions to consider when doing an M&E

Introduction to monitoring and evaluation Ungaluk Program 2015

Leadership & Management

Transcript of Introduction to monitoring and evaluation Ungaluk Program 2015