1

Workshop Monitoring a evaluace lidskoprávních projektů

Monitoring and evaluation of human rights projects

Inka Píbilová inka@evaluace.com

3. prosince 2013

2

Monitoring x evaluation x audit

Evaluation • Assessment of project efficiency, effectiveness, impact, relevance and sustainability for the purpose of learning and accountability to stakeholders

Monitoring • Ongoing analysis of project progress towards achieving planned results with the purpose of improving management decision making

Audit • Assessment of (i) the legality and regularity of project expenditure and income i.e. compliance with laws and regulations and with applicable contractual rules and criteria; (ii) whether project funds have been used efficiently and economically i.e. in accordance with sound financial management;; and (iii) whether project funds have been used effectively i.e. for purposes intended.• Primarily a financial and financial management focus, with the focus of effectiveness being on project results.

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

3

Why to monitor and evaluate projects?

• Accountability

• Learning

4

Project cycle management

Programming

Identification

FormulationFinancing

Implementation

Evaluation

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

5

Stakeholder Analysis Matrix

Example: River Pollution

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

6

Stakeholder Analysis – Power / Interest / Attitude In

fluen

ce/P

ower

KEEP SATISFIED INFLUENCE & ENGAGE

KEEP INFORMEDMONITOR

Interest

Policy makers

CSOsActivists

Attitude

7

Problem Tree

Example: River Pollution

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

8

Objectives and strategy selection

Example: River Pollution

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

9

Inputs – Activities - Outputs – Outcomes – Impacts

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Direct influence of implementers:Inputs I need for my work… (funds, people, time)Activities I do….

(research, conference)Outputs I produce …

(research paper, 100

participants to

conference)

Indirect influence of implementers:Outcomes I strive to…(Results) (raise awareness on X)Impacts I aim to…

(reduce human rights

violations)

10

Logical framework

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

11

Logical framework

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Issues identified during project evaluations:• Logical frameworks too complicated, too many objectives / indicators• Too ambitious and/or too vague objectives• Indicators difficult to measure / data not collected or difficult to collect• Logframe not used for internal monitoring / evaluation and decision

making

Tips:• Numbering Objectives (1,2), Results (1.1., 1.2., ...), Activities (1.1.1.,

1.1.2.)• Check source of verification – how will data be collected, by whom,

when/how regularly?

12

Quantitative - SMARTERSpecific / Simple (to understand, collect)MeasurableAttainable/Available at cceptable costsRelevant to project / stakeholdersTime-boundEvaluate/EngagingReevaluate/Recordable

Use both quantitative and qualitative indicators

http://www.smarttoolkit.net/?q=node/391http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Qualitative - SPICEDSubjectiveParticipatoryInterpreted and communicableCross-checked and comparedEmpoweringDiverse / disaggregated (by gender)

Compare using trends (increase), thresholds (min. 30%), targets (strategy by 12/Y1)

Min. 30 % of participants initiate a project aiming to address a local issue.

Reasons why participants have (not) implemented a project to address a local issue

13

Risks Analysis

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Assump-tions in

logframe

14

Monitoring: Why, for whom, what, how

Why?• Timely identification of

successes and problems during project implementation

• Informed and timely decision making by project managers to support implementation

• Accountability for the resources used and results achieved

• Stakeholder awareness and participation

• The evaluation of project achievements and audit of activities and finances

15

Use logframe to prepare Monitoring Plans

Year 1Indicator / Country

CZ PL SK Total Note

15.000 copies of publications distributed (5.000 each)

90% 110% 100% 100% Achieved

120 teachers trained (40 each) 45 40 35 120 AchievedMin. 50% of teachers implement the new tools within 6 months

60% 40% 30% 40% Not achieved

• Use indicators from the logframe• Split by year and partner• Use plan vs. actual, % or trends over time• Develop adequate sources of verification• Distribution lists• Attendance sheets with contacts!• On-line feedback forms after 6 months (combined with

supervision of teachers)• Media monitoring, public surveys...

16

Monitoring: Activity schedule with tasks and responsibilities

17

Monitoring Progress Report (Narrative)

http://www.europa.eu.int/comm/europeaid/qsm/index_en.htm

Monitoring Visit Checklist

18

Other monitoring tools

Forms at docs.google.com Geolocation at www.googlemaps.com, www.ushahidi.com

Monitoring tips

• Keep the users of information clearly in mind (who needs what information?)

• Build on local information systems and sources - costs and sustainability of new systems

• Collect only the minimum amount of information required – keep it simple and practical, more information is not better information

• Use Inception reports when team differs from the proposal writters, or when time passed by between identification and grants approval... usually 3 months after project start, revises original project proposal - updates logframe, sets annual action plan and monitoring plan

20

Evaluation: Why, for whom, what, how

21

Plan external evaluation well aheadTe

rms

of R

efer

ence

– O

bjec

tives

, sc

ope,

stak

ehol

ders

, que

stion

s,

budg

et, s

ched

ule,

out

puts

, use

.

Desk study Interviews

Surveys

Focus groups

Case studies Prel

imin

ary

findi

ngs

&

conc

lusi

ons

Finaldebriefing of all partners

Communication with the Project Partners

Draft evaluation report commented by all partners Fi

nal e

valu

ation

repo

rt

Inception phase

1-3 months

Field research

1-3 months

Reporting phase

1-2 months

Initi

al b

riefin

g an

d in

cepti

on

Evaluator Selection

22

Evaluation criteria

23

EC Evaluation Criteria

24

How to measure learning outcomes?

http://leanlearning.wikispaces.com/learning_analytics

25

Evaluation tools

26

What does participation mean?

27

Reporting: Why, for whom, what, how

28

Different evaluation methods and approacheshttp://betterevaluation.org/

Where to learn more

Road to Resultshttps://openknowledge.worldbank.org/bitstream/handle/10986/2699/52678.pdf?sequence=1

Evaluations of Czech development and humanitarian projectshttp://www.mzv.cz/jnp/cz/zahranicni_vztahy/rozvojova_spoluprace/dvoustranna_zrs_cr/evaluace/index.html

EPDET - annual training in Evaluations by World Bank consultantshttp://www.dww.cz/english.php?page=epdet1

EC Project Cycle Management Manualhttp://www.europa.eu.int/comm/europeaid/qsm/index_en.htm