Post on 13-Apr-2017
fiinovationOpportunity Solutions, Simplified.
MONITORING &EVALUATIONFRAMEWORK
IndexMonitoring and Evaluation & it’s importance1
Monitoring and Evaluation framework in the Project Program Cycle02
Monitoring and the inclusion of RBM03
Types of Monitoring04
Tailor-Made Evaluation Studies06
Evaluation Questions & RBM08
Six key steps for M&E Planning09
Minimizing Bias and Errors10
CSR in India - Industry Overview11
Fiinovation Expertise12
Some of our Initiatives 13
Organization Brief14
Awards & Recognition15
What Does Monitoring and Evaluation Mean?
Monitoring Evaluation
Why is Monitoring and Evaluation Important?Organisations use monitoring and evaluation for two key purposes:
01
Does your projecthave an M&E plan?
Does your projecthave an M&E binder?
1
Have you developed all monitoring forms?
3
Are monitoring data regularly used duringM&E meetings or other events?
8
Are staff and partners trained on data entry and analysis?
7
Are staff and partners trained onusing the monitoring forms?
4
Have you created amonitoring database?
6
Have you conducted a qualitycheck of the monitoring data?
5
If your answer is to any of the questions above, the project team should work to complete the ‘No'step(s). This review tool has not been designed to be used in isolation and refers to the package of practices being offered at Fiinovation, and this handbook is for further guidance at each review stage.
To learn about their own activities and results, and to support internal planning and development
To be accountable to their stakeholders.
Monitoring is a routine and systematic collection of information against a plan. The information might be about activities, products or services, users, or about outside factors affecting the organisation or project.
Evaluation is about making judgements about the value of any component part of an organisation
or its products, services or benets, or about the organisation as a whole.
Organisations need evidence of their efciency and effectiveness for funders, commissioners and investors. They also need to communicate achievements to a wider public. Charities are now specically required to report the achievements of the charity over the year. However, the role of monitoring and evaluation is providing learning and improving the organisation is of equal importance.
The following eight questions shall help in creating a better understanding on the need of monitoring and evaluation:
2
Monitoring & Evaluation
M&E Framework in the Project Program Cycle
02
Initial NeedsAssessment
Log frame and indicators
M&E planning
Baseline assessment
Midterm evaluation
and/or reviews
Final evaluation
Dissemination and use of
lessons
This is done to determine whether a project/program is needed and, if so, to inform its planning.
This involves the operational design of the project/program and its objectives, indicators, means of verification and assumptions
This is the practical planning for the project/program to monitor and evaluate the log frame's objectives and indicators.
This is the measurement of the initial conditions (appropriate indicators) before the start of a project/program.These are important reflection events
to assess and inform ongoing project/program implementation.
This occurs after project/program completion to assess how well the project/program achieved its intended objectives and what difference this has made.
This informs ongoing programming. Although this has been put in the last section of the diagram but, reporting, reflection and learning should occur throughout the whole project/program cycle.
There is no magic 10-percent sampling rule. It is important to note that the sample size is not related to the size of the popula�on being sampled. A frequent mistake is to conduct surveys among 10 percent of a given popula�on; in fact, it is likely that 10 percent of the popula�on is either too many or too few households. With too many households, the survey is using excessive resources and �me; with too few households, the sample will not adequately represent the popula�on.
Monitoring & Evaluation
Monitoring and the Inclusion of RBM
ImpactŸ Measuring changes at impact level requires a longer time frame, and is therefore dealt with
by evaluation and not monitoring.
OutcomesŸ Are the outputs leading to the achievement of the outcomes?Ÿ Is there anything happening that should lead management to modify the operations
implementation plan?
Output
ActivitiesŸ Are activities leading to expected outputs?Ÿ Are activities being implemented on schedule and within the budget?
Inputs
Monitoring Questions and RBM
03
A mistake common for M&E systems is to rely solely on either observa�on data or par�cipant responses. Observa�on data alone does not provide an explana�on of prac�ces or behaviors and o�en requires large assump�ons on the part of the M&E team. Focus group data (an example of par�cipant responses) may not capture important prac�ces that par�cipants do not see as relevant and may record instead what par�cipants think data collec�on teams want to hear.
Result Based Management (RBM) is an approach to project/program management based on clearly dened results, and the methodologies and tools to measure and achieve them.
RBM supports better performance and greater accountability by applying a clear, logical framework to plan, manage and measure an intervention with a focus on the results you want to achieve.
By identifying in advance the intended results of a project/program and how we can measure their progress, we can better manage a project/program and determine whether a difference has genuinely been made for the people concerned.
Monitoring and evaluation (M&E) is a critical part of RBM. It forms the basis for clear and accurate reporting on the results achieved by an intervention (projector program).
In this way, information reporting is no longer a headache, but becomes an opportunity for critical analysis and organizational learning, informing decision-making and impact assessment.
Ÿ How do beneciaries feel about the work/intervention?Ÿ What is causing delays or unexpected results?
Ÿ Are nances, personnel and materials available on time and in the right quantities and quality?
Monitoring & Evaluation
It tracks the use of inputs and resources, the progress of activities and the delivery of outputs.
It examines how activities are delivered – the efciency in time and resources.
It is often conducted in conjunction with compliance monitoring and feeds into the evaluation of impact.
For example, a water and sanitation project may monitor that targeted households receive septic systems according to schedule.
Types of Monitoring
It tracks effects and impacts.
This is where monitoring merges with evaluation to determine if the project/program is on target towards its intended results (outputs, outcomes, impact).
It measures whether there may be any unintended impact (positive or negative).
For example, a psychosocial project may monitor that its community activities achieve the outputs that contribute to community resilience and ability to recover from a disaster.
1. Results Monitoring
2. Process (Activity) Monitoring
3. Compliance Monitoring
4. Context (Situation) Monitoring
04
It ensures compliance with donor regulations and expected results, grant and contract requirements, local governmental regulations and laws, and ethical standards.
For example, a shelter project may monitor that shelters adhere to agreed national and international safety standards in construction.
It tracks the , especially as it affects identied risks setting in which the project/program operatesand assumptions, but also any unexpected considerations that may arise.
It includes the eld as well as the larger political, institutional, funding, and policy context that affect the project/program.
For example, a project in a conict-prone area may monitor potential ghting that could not only affect project’s success but, endanger the project’s staff and volunteers.
Monitoring & Evaluation
It tracks beneciary perceptions of a project/program.
It includes beneciary satisfaction or complaints with the project/program, including their participation, treatment, access to resources and their overall experience of change.
Sometimes referred to as it often includes a stakeholder Beneciary Contact Monitoring (BCM),complaints and feedback mechanism.
It should take account of different population groups, as well as the perceptions of indirect beneciaries (e.g. community members not directly receiving a good or service).
For example, a cash-for work program assisting community members after a natural disaster may monitor how they feel about the selection of program participants, the payment of participants and the contribution the program is making to the community (e.g. are these equitable?).
6. Financial Monitoring
05
5. Beneficiary Monitoring
It tracks the sustainability, institutional development and capacity building in the project/program and with its partners.
It is often done in conjunction with the monitoring processes of the larger, implementing organization.
For example, a National Society's headquarters may use organizational monitoring to track communication and collaboration in project implementation among its branches and chapters.
7. Organizational Monitoring
It accounts for costs by input and activity within predened categories of expenditure.
It is often conducted in conjunction with compliance and process monitoring.
For example, a livelihood project implementing a series of micro-enterprises may monitor the money awarded and repaid, and ensure implementation is according to the budget and time frame.
Monitoring & Evaluation
Tailor-made Evaluation StudiesEvaluations conducted during the life of the project should provide actionable recommendations to improve the quality of the project in the time remaining. The final evaluation findings must be incorporated into the strategy design for subsequent programming. The evaluation exercises can be categorized under three broad heads as enumerated below.
According to evaluation timing According to who conducts theevaluation
According to evaluationtechnicality or methodology
Formative evaluations occur d u r i n g p r o j e c t / p r o g r a m implementation to improve per formance and assess compliance.
Internal or self-evaluations are conducted by those responsible f o r i m p l e m e n t i n g a project/program. They can be less expensive than external evaluations and help build staff capac i t y and ownersh ip . Howeve r, t he y may l a c k c r e d i b i l i t y w i t h c e r t a i n stakeholders, such as donors, as they are perceived as more subjective (biased or one-sided). These tend to be focused on learning lessons rather than demonstrating accountability.
Real-time evaluations (RTEs) a r e u n d e r t a k e n d u r i n g p r o j e c t / p r o g r a m implementation to provide i m m e d i a t e f e e d b a c k f o r mod i f icat ions to improve ongo ing imp lementat i on . Emphasis is on immediate lesson learning over impact evaluation or accountability.
Summative evaluations occur at the end of project/program implementation to assess effectiveness and impact.
Externa l o r i ndependent evaluations are conducted by evaluator(s) outside of the implementing team, lending it a degree of objectivity and often technical expertise. These tend to focus on accountability.
Meta-evaluations are used to assess the evaluation process itself. Some key uses of meta-eva luat ions inc lude: take inventory of evaluations to inform the selection of future evaluations; combine evaluation results; check compliance with evaluation policy and good practices; assess how well evaluations are disseminated and utilized for organizational learning and change, etc.
Midterm evaluations are formative in purpose and o c c u r m i d w a y t h r o u g h implementation.
Participatory evaluations are c o n d u c t e d w i t h t h e beneficiaries and other key stakeholders, and can be empowering, building their capac i t y, ownersh ip and support.
Thematic evaluations focus on one theme, such as gender or environment, typically across a number of projects, programs or the whole organization.
06Monitoring & Evaluation
According to evaluation timing According to who conducts theevaluation
According to evaluationtechnicality or methodology
Final evaluations are summative in purpose and are conducted (o ften externa l l y ) at the completion of project/program implementation to assess how well the project/ program a c h i e v e d i t s i n t e n d e d objectives.
Joint evaluations are conducted collaboratively by more than one implementing partner, and can he l p bu i l d consensus at different levels, credibility and joint support.
Cluster/sector evaluations focus on a set of related a c t i v i t i e s , p r o j e c t s o r programs, typically across sites and implemented by multiple organizations (e.g. National Societies, the United Nations and NGOs).
Ex -post e va l uat i ons a re conducted some time after implementation to assess long- term impact and sustainability.
Impact evaluations focus on the effect of a project/ program, rather than on its management and delivery. Therefore, they t y p i c a l l y o c c u r a f t e r project/program completion during a final evaluation or an ex-post evaluation. However, impact may be measured during p r o j e c t / p r o g r a m implementation during longer projects/programs and when feasible. It highlights some of the challenges in measuring impact.
Proper management of an evaluation is a critical element for its success. There are multiple resources to support evaluation management which identify the key criteria and standards that guide how we plan, commission, conduct, report on and utilize evaluations. The framework mentioned in the beginning is to be applied to all evaluation activities. It draws upon the best practices to ensure accurate and reliable evaluations that are credible with stakeholders.
07Monitoring & Evaluation
Evaluation Questions and RBM
08
Impact
Outcomes
OutputŸ Were the operations objective achieved? (effectiveness)Ÿ Did the outputs lead to intended outcome?
Activities
Inputs
Ÿ Are the benets likely to be maintained for an extended period after assistance ends? (sustainability)
Ÿ What changes did the project bring about?
Ÿ Were the operation’s objectives consistent with beneciaries’ needs and with agency policies? (relevance)
Ÿ Were there any unplanned or unintended changes.?
Ÿ Were activities implemented on schedule and within budget? (efciency)Ÿ Were outputs delivered economically?
Ÿ Were stocks of items available on time and in the right quantities and quality?
Monitoring & Evaluation
Six Key Steps for M&E Planning
Review the project/program's operational design (RBM/LFA)Identify key stakeholder informational needs and expectationsIdentify any M&E requirementsScope of major M&E events and functions
Identify the purposeand scope of the M&Esystem
1.
Develop an M&E plan tableAssess the availability of secondary dataDetermine the balance of quantitative and qualitative dataTriangulate data collection sources and methodsDetermine sampling requirementsPrepare for any surveysPrepare specic data collection methods/toolsEstablish stakeholder complaints and feedback mechanismsEstablish project/program staff/volunteer review mechanismsPlan for data managementUse an indicator tracking table (ITT)Use a risk log (table)
Plan for data collectionand management
2.
Develop a data analysis plan, identifying the: i. Purpose of data analysis ii. Frequency of data analysis iii. Responsibility for data analysis iv. Process for data analysis.
Follow the key data analysis stages: i. Data preparation ii. Data analysis (ndings and conclusions) iii. Data validation iv. Data presentation v. Recommendations and action planning.
Plan for data analysis
3.
Assess the projects/program's human resources capacity for M&EDetermine the extent of local participationDetermine the extent of outside expertiseDene roles and responsibilities at each level of the M&E system.Plan to manage project/program team's M&E activitiesIdentify M&E capacity-building requirements and opportunities
Plan for M&E humanresources and capacitybuilding
5.
Itemize M&E budget needs i.Human Resource ii.Capital Expenses
Incorporate M&E costs into the project/program budgetReview any donor budget requirements and contributionsPlan for cost contingency
Prepare the M&Ebudget
6.
Anticipate and plan for reporting: i. Needs/audience ii. Frequency iii. Formats iv. People responsible.
Plan for information utilization: i. Information dissemination ii. Decision-making and planning
Plan for informationreporting and utilization
4.
09Monitoring & Evaluation
Minimizing Bias and ErrorsMinimizing bias helps to increase accuracy and precision. Accuracy means that the data measures what it is intended to measure. For example, if you are trying to measure knowledge change following a training session, you would not just measure how many people were trained but also include some type of test of any knowledge change.
As much as we would like to eliminate bias and error in our measurements and information reporting, no research is completely without bias. Nevertheless, there are precautions that can be taken, and the first is to we encounter in our work:be familiar with the major types of bias
1. Selection Bias Selection bias results from poor selection of the sample population to measure/study. Also called design bias or sample error, it occurs when the people, place or time period measured is not representative of the larger population or condition being studied. It is a very important concept to understand because there is a tendency to study the most successful and/or convenient sites or populations to reach (which are often the same). For example, if data collection is done during a convenient time of the day, during the dry season or targets communities easily accessible near paved roads, it may not accurately represent the conditions being studied for the whole population.
2. Measurement Bias Measurement bias results from poor data measurement – either due to a fault in the data measurement instrument or the data collector. Sometimes the direct measurement may be done incorrectly, or the attitudes of the interviewer may influence how questions are asked and responses are recorded. For instance, household occupancy in a disaster response operation may be calculated incorrectly, or survey questions may be written in a way that biases the response, e.g. “Why do you like this project?” (rather than “What do you think of this project?”).
3. Processing Error Processing error results from the poor management of data – miscoded data, incorrect data entry, incorrect computer programming and inadequate checking. This source of error is particularly common with the entry of quantitative (statistical) data, for which specific practices and checks have been developed.
4. Analytical Bias Analytical bias results from the poor analysis of collected data. Different approaches to data analysis generate varying results e.g. the statistical methods employed, or how the data is separated and interpreted. A good practice to help reduce analytical bias is to carefully identify the rationale for the data analysis methods.
10Monitoring & Evaluation
1CSR Expenditure Among 10 Major Industries
HEALTH
22%RURAL DEVELOPMENT
33%
EDUCATION
23%
ENVIRONMENT
22%
2012-13
Expense on CSR in the Private and Public Sector
20 400 60 80 100
Slum Area Developent
RD Projects
Technical Incubators
PM Relief Fund
Sports
Forced Army Veterans
National Heritage
Environmental Sustainability
Reducing Inequalities
Education, Vocational Skills
Hunger, Poverty, Healthcare
955
6535
1882
8515
3070
955
5842
4258
6337
7525
2278
Pri
vate
Se
cto
rP
ub
lic
Se
cto
r
1 As per the annual reports of the top 200 firms for the year 2012-132 FICCI and Nextgen
CSR in India - Industry Overview
11Monitoring & Evaluation
Fiinovation ExpertiseA vast bank of expertise gained over the years across a wide range of sectors and research methodology. And also, the flexibility to serve a wide range of projects in the sectors of: Livelihood, Education, Health and Environment.
Defined M&E standards that help improve program quality and positively impact the people we serve. The standards followed at Fiinovation reflect the key characteristics of high-quality programs and agency culture that promotes better learning and strengthens accountability to stakeholders. These are critical elements of a high- performing, dynamic learning organization.
Triangulation of data is another key step practiced at Fiinovation, which involves collecting data from multiple sources, sometimes using multiple tools, to identify and reduce bias. If triangulation of qualitative data is not carried out, a risk of biasing or distorting the collected data exists, resulting in incorrect or incomplete information which might result in skewed understanding of the circumstances.
By collecting data from multiple sources or with multiple tools, we identify and address discrepancies or inconsistencies in the data. Triangulation often leads to additional questions or clarifications, which can be answered through follow-up interviews, discussions or exercises.
Core Values
We ensure excellent quality of the actionable reports which is intended to provide absolute and deta i l ed ins ight into the markets, products, competition and the overall perspective at an affordable price.
We ensure secrecy for all the research projects that we ca r ry ou t and ma i n t a i n confidentiality as per the code of conduct.
We guarantee authenticity, reliability and credibility of the data, facts and figures that we gather and d i ssem inate through primary and secondary research.
Aut
hent
icit
yQ
uality
Confidentiality
12Monitoring & Evaluation
13Monitoring & Evaluation
The Monitoring and Evaluation Team of Fiinovation has been involved in helping its partners to carry out eld research studies for monitoring, evaluating and assessing the impact of their proposed and ongoing/complete projects.
A few of our successfully completed assignments in the current year are mentioned below:
Fiinovation partnered with IL&FS Skills to carry out an integrated on-site assessment study for assessing the current status of the community in terms of their readiness and acceptability for skill based trainings. The study was carried out to explore the avenues that shall be available at the proposed site for Power Plant in Nana Layja (Mandvi), Gujarat. Based on the research ndings of Fiinovation, the project aims to develop skill, employment and entrepreneurial opportunities for the population of Mandvi Block in Kutch district with respect to the Nana Lajya Power Plant being put up by IL&FS Group. Fiinovation gathered adequate information through extensive research, both primary and secondary sources for the purpose of the study. The Need Assessment of the project was carried out for 3 months with assessment tools intrinsically focused on the need and acceptance of the community.
Fiinovation has also conducted a summative evaluation study of the skill training institute (Cairn Enterprise Center - CEC) of Cairn India Limited (CIL) at Barmer (Rajasthan). The study was carried out for the CSR project of CIL being implemented by IL&FS Skills with a view to evaluate the program performance and its effectiveness by collecting information that can be used for further evaluation. Fiinovation developed tailor-made evaluation parameters for assessing the outcomes of the skill training intervention. The framework used empirical evaluations aimed at elucidating and substantiating the linkages between the trainings provided and the intended or observed benets. The project required unbigoted opinion along with technical expertise which was provided by Fiinovation for assessment and recommendations for future course of action.
With its team of experts, Fiinovation has successfully conducted evaluation and assessment of Adult Literacy Centers (ALC) run by M/s JK Tyre & Industries Ltd. at Kankroli (Rajasthan), Morena( Madhya Pradesh) and Mysore( Karnataka). For this purpose, a detailed research was carried out to assess the efcacy and effectiveness of the ALCs.
Some of our
Initiatives
Afliations
Established in 2008, Fiinovation (Innovative Financial Advisors Pvt. Ltd.) has created a niche as a multi disciplinary research organization in Corporate Social Responsibility (CSR) and Sustainability domain. Headquartered in New Delhi, it has been forging partnerships between credible NGOs and corporations that are looking to implement and initiate community based interventions and initiatives. With a pool of over 4000 credible NGOs spread across the country, the motive has been to create awareness, inspire innovative thinking and action with sustainable development at its core. These NGOs are empanelled post a thorough and stringent due diligence check through secondary research and on ground evaluation.
Fiinovation is known in the development sector for enhancing quality across organizational value chain through its in-house Proposal Design & Research Laboratory for designing innovative CSR initiatives. Fiinovation also supports corporations through conducting need assessment, monitoring and evaluation as well as impact assessment of their programs.
The distinct practices of Fiinovation include CSR Portfolio Management (CPM), CSR-CSO Partnership, Initiative Design, Initiative Management, Monitoring & Evaluation, Impact Assessment and CSR audit across verticals of health, environment, education and livelihood, which contribute to INR 15 crores in the top line of nancial statement.
Fiinovation has been a knowledge partner with various industrial bodies such as Confederation of Indian Industries (CII) and Institute of Directors (IOD). With CII, the partnership emphasizes on research for promotion of afrmative action along with CSR and social enterprises. The organization has deliberated with various industrial leaders at numerous conferences conducted on corporate governance, environment management and CSR & sustainability during its path breaking partnership with IOD. Being a research organisation, Fiinovation also contributes to IOD's Master class for Directors that has a wide spectrum of subjects such as corporate governance, sustainability, HR practices, nance, corporate social responsibility, environment management and board effectiveness. Other than this, the organization has been closely associated with other forums such as NGO BOX and World CSR Congress.
Organization Brief
14Monitoring & Evaluation
Awards & Recognitions
Mr. S. Chakraborty along with Smt. Sheila Dixit at the 24th World Congress on Total Quality and Leadership
15
Fiinovation has been conferred with the “CSR Team of the Year” & “Caring Company Award” at the World CSR Day 2015. It was also the recipient of the 'Best Social Innovation: Product or Service' at the World CSR Day 2014. Apart from this, the company was bestowed with the "Manager of the Year" & "Best Enterprise of the Year - in the eld of Health, Education, Environment and Livelihood with a focus on CSR & Sustainability" Awards by the European Business Assembly at the Socrates Award Ceremony, Summit of Leaders in 2014 at Oxford Town Hall, United Kingdom.
Mr. Soumitro Chakraborty addressing the gathering at CII conference - Bridging the Gap: Fostering growth through Education & Entrepreneurship
Monitoring & Evaluation
Media Links
fiinovation
24/30, Ground Floor, Okhla Industrial Estate, Phase III, New Delhi - 110020, IndiaPhone: 011-42332200 | Fax : 011-42332205Email : info@inovation.co.in, | Website : www.inovation.co.in