Organized by Small Scale & Micro Irrigation Support...
Transcript of Organized by Small Scale & Micro Irrigation Support...
0
Workshop on Developing M&E system for Oromia Irrigation Development Authority
February 5-6, 2016 Adama, Ethiopia
Organized by
Small Scale & Micro Irrigation Support (SMIS) Project – Oromia Regional Office
In collaboration with Oromia Irrigation Development Authority (OIDA)
Workshop Proceedings and Report
Prepared By
Regional M&E Expert
In association with:
1 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Table of Contents
1. Background: ............................................................................................................................... 2
2. Objectives of the Workshop: ..................................................................................................... 2
3. Workshop Participants: ............................................................................................................. 3
4. The Workshop Proceedings: ...................................................................................................... 3
4.1 Proceedings of Day One: ........................................................................................................ 3
4.2 Proceedings of Day Two: ....................................................................................................... 8
5. Next steps/ way forward ........................................................................................................... 8
6. Concluding Remarks .................................................................................................................. 9
7. Closing Remarks ......................................................................................................................... 9
8. Conclusion ............................................................................................................................... 16
9. Recommendation .................................................................................................................... 16
10. Annexes .................................................................................................................................. 17
2 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
1. Background:
Small-Scale and Micro Irrigation Support (SMIS) Project is aims to support the four major regions to
develop the capacity of public Institutions in alignment of the Small Scale & Micro Irrigation capacity
Development Strategy. One of the area of those capacity development, which is under the direct
influence of the project, is improving the capacity of public institutions to coordinate and manage
the planning, design and construction of SSI schemes. Specifically, Monitoring & Evaluation (M&E)
is one of the area in which the project is intended to provide its support. As it is denoted on the
project year one (PY1) Annual Work Plan (AWP), the Work Break down Structure (WBS) 1110.2 is
about providing support to Oromia Irrigation Development Authority OIDA in producing M&E
template.
In line with this, to assess and identify gaps and need, SMIS Oromia Regional Office has been
conducting consecutive meetings with the Leader (Ato Samuel, deputy head), Managers and experts
of OIDA. Based on the identified gap and need, several forums have been conducted:
1. to identify information needs by the three core processes of OIDA (Extension, Study, Design
& Construction and Scheme Administration) that is going to be provided by the M&E system;
2. to develop institutional Logic Model (LM) based on the identified information needs.
As a result, in collaboration with OIDA, OSMS has proposed and developed this workshop to work
on the following areas of the M&E system development steps.
2. Objectives of the Workshop:
To define indicators for each results statement mentioned in the LM.
To prepare the first version of the M&E Framework.
To discuss on the identified actions and way forward to have a full package of the M&E
system
To discuss and reach on consensus on the way forward
3 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
3. Workshop Participants:
One Toolkit was issued to each participant on arrival at the workshop to capture number
of participants and positions held by participants as provided in Annex I. There were 20
participants at the workshop (Annex II), reflecting a wide range of representation from the
three process owners of OIDA, Small Scale Irrigation Directorate (SSID) of the Ministry of
Agriculture (MoA) and Oromia Bureau of Finance and Economic Development (BoFED).
4. The Workshop Proceedings:
4.1 Proceedings of Day One:
The Opening Session:
The workshop was opened by speeches from OIDA Ob. Kifle Bekele, Extension Process Owner.
Key Messages from the Opening Session:
The objective of the workshop is to refine the LM, to identify indicators for results
statement in the LM and to develop an 8 column M&E Framework.
4 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
M&E system is a management tool
The initiative to develop the M&E system for OIDA is demand driven
The system is to reinforce the current reform tool tha is being applied i.e Balanced Score
Card (BSC)
SMIS is our key partner which follows-up our activities
The contribution of the participants during the duration of the workshop is highly valuable
Having pointed out the above highlights, Ato Kifle declared that the workshop is officially
opened.
Following the opening session, the presenter, Mr. Robin, International M&E Expert, asked the
participants about their understanding / background on Result Based Management (RBM). Their
reflection is summarized as follows:
No background
Have back ground but no application
Have background on Log frame
The presenter reflected that weather they have a background or not, they have already developed
the LM and it indicates that it is not a complex issue. Then after, he explained the objectives of the
workshop and the expected outcome as stated above in the key messages. In addition, he has also
marked on the schedule for the two days.
The schedule:
Presentation on overview of Result Based M&E
Presentation on OIDA draft LM
Presentation on Indicators
Individual and Group Work on Developing Indicators
Plenary Discussion to Finalize Indicators
Next steps to put in place the M&E system
Three presentations were made on the first day including RBM, LM and Indicators Presentation on RBM presented by Mr. Robin D’Arcy, International M&E expert,
He highlighted the following points:
5 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Goal of RBM is to create an easy system for planning results and outcomes we want to
achieve
Based on the LM we plan activities
Key to RBM- achievement of results are different from activities completed
We can measure activities and changes
RBM assists to know whether achieving the goals
If there is weakness, need to rectify the challenge by asking why? We can change
programs if something didn’t work.
The second presentation on OIDA draft LM presented by Ob. Sunyom, M&E Expert.
.
6 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
The key points in the presentation:
LM is a planning tool, one of the tools of RBM
Its development starts from the higher level by answering “How” to get into lower
level
LM is designed from the goal and mandates of OIDA.
To achieve the goal, OIDA has to accomplish several activities, produce deliverables
and achieve immediate results.
Based on this OIDA you have developed an LM already
Developing LM is subject for improvement as it is not a one time activity
The International M&E expert asked the participants if they have any reflection / comment to
further refine the draft LM. Some points has been raised from participants. As a result, participants
have commented that “community mobilization” to be included as an activity. The international
M&E expert also asked the participants “does the LM captures what OIDA do?” participants reply
“ yes it captures” (Annex III OIDA LM)
The third presentation on Indicator made by Mr. Robin D’Arcy, International M&E expert,
Key Points in the presentation
It is a key element in PMF/ M&E framework to figure out weather we are achieving our
targets.
Good indicator is a tool to inform “How do we measure our organization success?”
We have to identify indicators that we can control
Output indicators measure the results of our activities (what we have done, “how many
people trained”) while outcome indicators measure changes we achieve as a result of our
activities (How well / successfully we have done our job.
Indicators should be mixed – quantitative and qualitative.
Indicators should consider gender disaggregation.
After the presentation, Individual exercise on indicator setting has been conducted. Following the
exercise, four thematic group were formed to undertake practical identification of indicators for
the 9 outputs, 5 immediate outcomes and an intermediate outcome stated in the LM and 1st
column of the PMF of OIDA
.
7 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Working Groups on Day One: Group One: Study, design and construction thematic group to develop indicators for results stated under WBS 1000 and their respective outputs 1100, 1200, 1300 & 1400. Group two: extension thematic group to develop indicators for WBS 2000 and 4000 and their Corresponding outputs 2100,4100 & 4200. Group Three: Irrigation Agronomy thematic to develop indicators for WBS 3000 and their particular outputs 3100. Group four: Linkage thematic (for multi-stakeholder’s involvement) to develop indicators for Intermediate Outcome and Immediate outcome stated under 5000 and their outputs 5100.
Several indicators had been identified and incorporated in one Framework. Mr. Robin has commented that indicators are chosen properly.
8 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
4.2 Proceedings of Day Two:
The indicators in the framework were presented to plenary by Mr. Robin. Plenary discussion, debate
and revision continued. Thematic debate on irrigation agronomy vs irrigation extension was raised to
differentiate roles among them. The participants had agreed that agronomy is about agronomic
technology package development while extension is a communication tool to suitably deliver those
technologies to the grass root level. Accordingly, indicators setting for the two themes became
somewhat clear.
Findings of the Group Work After a long discussion and debates the four groups managed to identify 51 (4 for one Intermediate,
14 for 5 Immediate and 33 for 9 output level results (Annex 2: OIDA M&E Framework).
After having a bunch of indicators, the next activity was developing the Framework which is mainly
to incorporate how, from where/who and how frequent and by whom to collect the data on the
indicators. The previous groups reconvened and develop the remaining column of the framework
with the above stated variables. The information for the framework was collected and summarized
by Mr. Robin.
Findings of the Group Work
In the end, a draft M&E framework is developed (annex IV) to be approved by OIDA Leaders after
refined and finalized with full collaboration with OIDA managers and technical staff.
5. Next steps/ way forward
The next agenda was discussing on the next steps/ way forward to have a comprehensive M&E system. Summary of the steps are as follows:
1. Finalize the Framework 2. Develop Data Collection Tools 3. Develop M&E system Manual 4. Develop Database 5. Train Enumerators
9 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
6. Implementing the M&E system /Baseline survey/ 7. Results Reporting 8. Lessons learnt (what has the results? Why? To be changed if needed
6. Concluding Remarks Closing remark was mad by Ato Yohannis Geleta RTL – OSMIS with the following points:
This Workshop is designed as part of capacity development activity this is being
undertaken by OSMIS. We are moving 10% forward and we need to exert a 90% effort
to achieve our goal- developing M&E system and implementation
After having a complete M&E Framework, soon we have to develop data collection
tools. To host the to be collected, database is required to set up at OIDA. We need to
identify the suitable one depending on contexts either server based /internal/, web
based, SQL or MS offices (Excel or Access). Soon we will provide SPSS training for PIs
which will create noble opportunity for data analysis and reporting skills. For detail
steps forward refer Annex V.
Resources /capacities and capabilities/ to implement the system proposed to get from
OIDA, OSMIS, and OIDA AGP.
Finally, Ato Yohannis invited all process owners to reflect on the whole process of the
way forward. In one vote, all of them have expressed their consent.
7. Closing Remarks
Closing remark was made by Ato Kifle, Extension Process owner- OIDA. The points raised as bellow:
He extended his thanks to OSMIS for bringing such a very important issue on board. One of OIDA’s Problem is lacking the M&E system to measure precisely what OIDA is doing. OIDA doesn’t know where it is going but it is doing a number of activities. Such initiative can be scaled out to other institutions too.
He also thanked OIDA and SMIS for the inputs they exerted in producing such valuable output and spending their precious time. He also reminded that only 10% of the required job has been done, therefore all of us have seriously work for the 90%.
10 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Feedback Provided by Participants
Feedback was provided via a workshop evaluation form completed by participants. A copy of this
form is provided as Annex VIII to this report. As there was no compulsion for attendees to complete
this form, not all participants completed these forms. Out of total attendees of 20, 18 of them
completed more or less all parts while, 2 of them didn’t complete the “facilitation skill effectiveness”
part at all.
Three types of feedback tools were prepared to capture the changes in participants’ confidence level
in their skills and knowledge before and after the Workshop; to get expectation level of participants
on the relevance and applicability of the workshop to their day to day works; and to receive
comments on the facilitation skills effectiveness. The tools include the following: -
Section A- Assessment of Learning- Knowledge & Skills gained
Section B- Relevance, Effectiveness & Applicability of the Discussion points/forum Section
Section C- Facilitation skills effectiveness
The results of the evaluation have therefore been calculated accordingly and presented as below
A. Assessment of Changes of skills and knowledge (before & After)
Rating Before
Question
Rating After
1 2 3 4 5 AV. 1 2 3 4 5 AV.
10 4 3 2 1 2 A.1 I understand how M&E framework is established
(K). 0 0 4 8 8 4.2
6 7 4 2 1 2.25 A.2 I know the relationship between results
statement and Indicators (K). 0 0 4 9 7 4.15
6 7 5 1 1 2.2 A.3 I can identify information needs/ indicators for
results statement(S). 0 0 5 6 9 4.2
9 7 3 1 1.8 A.4 I can design an M&E framework for SSI and MI
projects (S) 0 0 7 8 5 3.9
7 5 6 1 1 2.2 A.5 I understand the role of M&E framework (K) 0 0
2 10 8 4.3
2.09 Average 4.15
41.8% Percentage 83%
Net Gain 2.06
41.2%
11 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
The participants rated the workshop as Strongly Agree [5], Agree [4], Neither agree or Disagree [3] or Disagree
[2] and Strongly Disagree (1). A maximum rating possible was therefore a rating of 5 while a minimum rating
was 3. The average of all ratings was 4.15 or 83%. Compared to before the workshop rating which was on
average 2.09 or 41.8%, there is a net gain of 2.06 or 41.2% => almost double. The theory of Adult learning
states that the retention rate of KS from immediate use of learning strategy is 90%. Therefore, this result
supports the theory.
It can be seen from the graph that, in all the five question, the average KS level after the workshop
has been doubled compared to the level before.
22.25 2.2
1.82.2
4.2 4.15 4.23.9
4.3
2.21.9 2 2.1 2.1
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
Q1 Q2 Q3 Q4 Q5
Change in level of Knowlede and Skills (1)
Av Level of KS Before Av. level of KS After Av.Net effect Gained
12 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
The graph above shows 69% of the participants rated the level of their KS improvement as strongly
agree or agree. While no participant is found in a rating scale of strongly disagree or disagree to
their skill level while 68% of them were in this rating before the workshop.
Feedback A6: Valuable Aspects/ most useful
Here the participant was asked to identify what they perceived to be the most useful aspect of the
workshop. Although there were different styles of response, only 3 respondents answered this one
with the thee responses being in open text format:
Compare to KS gained from other similar trainings I took abroad, this one will help me to my
work. Both the LM and M&E FW are most useful. (one response)
I get KS on how to build the M&E framework (one response)
The RBM aspect is most useful that focusing to evaluate results rather than activities (one
response)
The three responses provided accounted for 3 out of 20 responses or equivalent to almost 15%
of all responses. The response for the 17 are nil.
38%
30%
21%
7%4%
0 0
23%
35% 34%
0%5%
10%15%20%25%30%35%40%
StronglyDisagree
disagree Neutral Agree StronglyAgree
Change in Level of Knowledge and Skill (2)
Before After
13 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Section B: Relevance (R) and Applicability (A) of the workshop to Participant Learning Needs. Effectiveness (E) of the Workshop Methodology
Question on Relevance and Applicability of this Event to My Work Effectiveness of the Discussion Forum Methodology
Rate
1 2 3 4 5 Av.
B.1 I understand how the content of the workshop is relevant to my work (R)
5 15
4.75
B.2 I can apply what I learned to my work. (A) 5 15 4.75
B.3 I will refer to the Logic model and the M&E framework at my work. (R) 13 7 4.35
B.4 The process of the workshop (participatory, learner-centered, and experiential) helped me to better understand the content. (E)
8 12
4.6
B.5 I would recommend this training to my colleagues. (R) , (A) 5 15 4.75
Average 4.64
Here the participants rated the extent to which the workshop met participant expectations by
selecting a rating from Strongly Agree [5], Agree [4], Neither agree or Disagree [3] or Disagree [2]
and Strongly Disagree (1). A maximum rating possible was therefore a rating of 5. The average of all
ratings was 4.64 or 92.8%.
100% of the participants are strongly agree and agree that the content in the entire process of the
system development is relevant and applicable to their work. The graph above shows the proportion
of respondents who rated the content of the workshop as agree or strongly agree.
Agree (36%)
Strongly agree (64%)
Proportion of participants rate on relevance and applicability of the workshop content
14 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Feedback B6: Workshop approach liked:
Here the participant was asked to identify the workshop approach/ methodology they like.
Responses were provided in open text format. The response is 50%: 10 responses out of 20 received.
The comments received centered on:
Participatory:
Group discussion
Open discussion
Group Exercises
Workshop materials
Individual exercise
All methods
Feedback B7: Workshop approach did not liked:
Here the participant was asked to identify the workshop approach/ methodology they don’t like.
Responses were provided in open text format. The response is 35%: 7 responses out of 20 received.
The few comments received centered on:
Nothing to dislike: 3respondents
Discussion which abridged time for facilitators: 2 respondents
Was not participatory: 1 respondent
Trainers allowed side talk: 1 respondent
Feedback B7: opportunities to improve the workshop next time:
Here the participant was asked to suggest as to how the workshop could be improved. Responses
were provided in open text format. The response is 45%: 9 responses out of 20 received.
The comments were centered:
In application: 1 respondent
Allocate enough time, minimum 5 days: 5 respondents
RBM is new, so prior workshop on RMB required: 1 respondent
Participate other stakeholders: 1 respondent
Participate irrigation related institutions: 1 respondent
15 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
C. Feedback on Facilitation Skill Effectiveness
Delivery of Discussion Forum
Rate
1 2 3 4 5 Av.
C.1 There was enough time for group work, group presentation and discussion.
5 2 5 6 3.7
C.2 Everyone, including me, was able to participate freely. 1 7 10 4.5
C.3 The process was guided in a logical way so I could see how we were proceeding to meet our learning objectives.
6 12 4.7
C.4 Questions were answered with helpful examples. 4 14 4.8
C.5 Varied presentation techniques were used to keep the sessions interesting.
1 4 13 4.7
C.6 The workshop venue provided an appropriate and comfortable learning environment.
2 16 4.9
Average 4.5
Here the participants rated five [5] aspects of presenters’ performance with a rating from Strongly
Agree [5], Agree [4], Neither agree or Disagree [3] or Disagree [2] and Strongly Disagree (1). A
maximum rating possible was therefore a rating of 5 for each aspect.
The six aspects rated were:
o Enough time provision for group work, presentation and discussion: The average of all
ratings was 3.7 or 61.7%
o Participation freedom: The average of all ratings was 4.5 or 75%
o The process was guided by logical way: The average of all ratings was 4.7 or 78.3%
o Questions were answered with helpful examples: The average of all ratings was 4.8 or 80%
o Varied presentation techniques: The average of all ratings was 4.7 or 78.3%
o Workshop Venue: The average of all ratings was 4.9 or 81.7
Lesson: individual comments centered on 'not enough’ time
Feedback C7: something to be done for better event:
Here the participant was asked advise on what to be done to make such an event better. Responses were provided in open text format. The response is 45%: 7 responses out of 20 received. The few comments received centered on:
Collaborative action: 1 respondent
16 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Better if we participate before training: 1 respondent It is Best: 1 respondent Participate different stakeholders on baseline data collection: 1 respondent Time allocated must be extended to optimum level: 2 respondent Work with other stakeholders: 1 respondent
8. Conclusion
The Workshop objective was to provide our key stakeholder-OIDAs senior managers and thematic
area experts with the knowledge and skills necessary to design M&E framework. The workshop
included and brief training on overview of result based M&E system, mentoring and coaching the
indicator development and data collection methodologies process.
As per the feedback form the participants, the learning process was interactive, participatory and
well facilitated. As a result, participants highly contributed their tacit and explicit K&S for the
workshop objective to be realized with significant success. A bunch of draft indicators at all levels of
mapped outcomes in the LM and data collection methodologies has been suggested. There were
useful debates and discussion to reach on these outcomes (indicators and data collection
methodologies). There was significant evidence in the participant feedback forms, that participants
appreciated the value of the workshops to their work. There was also positive feedback that the
workshop time should be extended to optimum level. During the introduction session made by Mr.
Robin, most participants expressed that they were not familiar with RBM.
From the process, individual and collective learning were initiated. Accordingly, mutual
understanding was reached on the way forward to put in place the comprehensive M&E system for
OIDA
9. Recommendation
SMIS has to continue facilitating the process to nurture the capacity of OIDA to finalize
the work
To deepen the participant’s understanding on RBM, inclusive training on RBM is
important
17 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
10. Annexes
Annex (I): Workshop Agenda
Friday February 5, 2016 8:30 - 8:45: Introductions 8:45 - 9:00: Workshop overview 9:00 - 9:30: Overview of Logic Model 9:30 – 10:30: Presentation on M&E framework development 10:30 – 10:45: Break 10:45 – 12:30: Presentation on M&E framework development 12:30 – 14:00: Lunch 14:00 – 15:30 Indicator development 15:30 – 15:45: Break 15:45 – 17:00: Presentation and discussion Saturday February 6, 2016 8:30 – 10:30: Revision and finalization of indicators 10:30 – 10:45: Break 10:45 – 12:30: M&E framework development 12:30 – 14:00: Lunch 14:00 – 15:00: Presentation/discussion of M&E framework 15:00 – 16:45: Finalization of M&E framework 15:45 – 16:00: Wrap up and next steps Annex (II): List of Participants
S.# Name of Participant Sex Institution Position Mobile Address
E-mail Address
1 Mohammed Hassen M OBoFED Expert 912026110 [email protected]
2 Mulugeta Melese M MoA Expert 912307919 [email protected]
3 Gobena Chemeda M OIDA E.M Expert 921071341 [email protected]
4 Girma Abebe M OIDA Sociologist 911573884 [email protected]
5 Sheleme Beyera M OIDA Agronomist 913694254 [email protected]
6 Firaol Muluneh F OIDA Engineer 912135622 [email protected]
7 Kifle bekele M OIDA Extension core Process owner
911853032 [email protected]
8 Tamirat Mengistu M OIDA Process owner-scheme admin
912398622 [email protected]
9 Ketema Tsfaye M OIDA Study & design Process owner
912260276 [email protected]
10 Sunyom Garomsa M OIDA M&E expert 911096828 [email protected]
11 Solomon Bekele M OIDA Planning, M&E Process owner
911675337 [email protected]
18 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
S.# Name of Participant Sex Institution Position Mobile Address
E-mail Address
12 Dagnachew Delesssa M OIDA-AGP coordinator 911995537 [email protected]
13 Tamirat Assefa M OIDA community Participation Process owner
912688365 [email protected]
14 Tewodros Gebeyehu M OIDA Economist 912233138 [email protected]
15 Abebe Merga M OIDA Planner 911751785 [email protected]
16 Yaddessa Tesfaye M E/Wollega IDA
Expert 917039031 [email protected]
17 Nahimi Belay M E/Wollega IDA
Economist 911027857 [email protected]
18 Birhanu Maru M OIDA Horticulturalist 912330828 [email protected]
19 Debebe Bayuu M OIDA Environmentalist 911894245 [email protected]
20 Zegeye Kassahun M MoA-NR Agronomist 937283984 [email protected]
Annex (III): Logic Model
Adobe Acrobat
Document
Annex (IV): Draft M&E Framework
OIDA Draft M&E
Framework Feb 12 (1).docx
Annex (V): Plan of Action for next steps
OIDA M&E system
development activity overview.docx
19 | P a g e Workshop on M&E system Development, Adama , February 5-6, 2016
Annex (VI): OIDA M&E System work Plan Gantt Chart (next steps)
OIDA M&E system
workplan gantt.xlsx
Annex VII Data collection tools to be developed
Requried data
collection tools.docx
Annex VIII Participants feed back form
Workshop
Evaluation form.doc