Slide 1

Slide 2 Constructing a Monitoring & Evaluation Plan Presented by: Cindy Banyai, Ph.D. Executive Director, efocus Institute IREX Community Solutions Program September 8 September 15, 2011 Slide 3 Learning Objectives Identify key steps in constructing an M&E plan Determining an evaluation schedule Building off the logframe Selecting indicators Designing evaluation questions Using evaluation data 2 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 4 SIGNIFICANCE OF EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 3 Slide 5 Evaluation is important because it provides valuable information to stakeholders and administrators reduces waste and inefficiency improves communication between important parts of a policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 4 Slide 6 Why evaluate? Accountability Disclose findings to stakeholders Use findings to inform resource allocation and other decisions Metrics matched to meaningful outputs and outcomes under the control of responsible organization Compare performance to ex ante commitments and targets Use methods that obtain internal validity of measurement Ensure credibility of analysis 5 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 7 Why evaluate? Learning Generate knowledge about magnitude and determinants of project performance Leading project/program designers and implementer to refine designs and introduce improvements into future efforts Requires careful selection of evaluation questions to test fundamental assumptions underlying project designs, methods that generate findings that are internally and externally valid Develop systems to share findings widely and facilitate integration of the evaluation conclusions and recommendations into decision-making (USAID 2011) 6 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 8 Evaluation Schedule Determine when all evaluation activities will occur throughout the PMC Monitoring activities must occur at regular intervals (yearly, quarterly, bi-annually) depending on timeframe of policy/program/project 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 7 Slide 9 Project Management Cycle (PMC) 8 Based on Miyoshi, 2008 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 10 STEPS IN EVALUATION 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 9 Slide 11 Steps in evaluation 1. Confirm purpose of evaluation 2. Organize information 3. Design evaluation work plan 4. Collect data 5.Analyze and interpret data 6. Report evaluation results 7. Give feedback 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 10 Slide 12 1 - Confirm purpose of evaluation What is the evaluation objective DAC 5? What is the purpose of the evaluation results? Who are the potential users of the evaluation results? 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 11 Slide 13 DAC 5 Criteria and logic model 12 Miyoshi, 2008 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 14 2 - Organize information Gather materials on policy/program/project to be evaluated Identify necessary documents, informants 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 13 Slide 15 3 - Design evaluation work plan Select indicators Determine evaluation questions, basis for judgment, data needed Choose appropriate evaluation approach and data collection methods Consider time, resources available 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 14 Slide 16 Indicators and Measurement Indicators To understand phenomena Must be direct and valid Measurement To make indicators measurable or operational Must be practical and reliable Examples Increase in knowledge: the average score of a nation-wide test Better living standards: increase in income 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 15 Slide 17 Selecting indicators SMART Specific Measureable Attainable Relevant Timely ( AusGUIDElines 2003) 16 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 18 Indicators must also be Adequate Represent objectives and avoid having too many indicators for only one purpose 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 17 Slide 19 Building off the logframe Use logframe to categorize and demonstrate connections between indicators 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 18 Slide 20 Logframe 19 Miyoshi, 2009; based on JICA 2004 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 21 Evaluation Questions Refer to supplement with question prompts based on DAC 5 Tailor to suit specific needs of your evaluation Follow supplemental worksheet 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 20 Slide 22 Types of evaluation questions 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 21 Miyoshi, 2008 Slide 23 22 Types of Evaluation & Evaluation Questions Ex-anteMid-termTerminalEx-post Normative Question: Performance measurement Baseline study Descriptive Question Process evaluation Cause effect relationship: DAC 5 Criteria Relevance Impact / Effectiveness/ Efficiency Sustainability Prediction Actual, Optional Miyoshi, 2008 Slide 24 4 - Collect data Gather information according to selected methods Qualitative, quantitative methods Surveys, interviews Participatory evaluation Focus groups, projects 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 23 Slide 25 5 - Analyze and interpret data Organize collected data Analyze results Process statistics from quantitative data Build narratives or code and render statistics of qualitative data Interpret analysis Answer evaluation questions Compare analysis to indictors and benchmarks Make recommendations, draw lessons from process 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 24 Slide 26 6 - Report evaluation results Put together analysis and interpretation of evaluation Convey to potential users 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 25 Slide 27 Reporting Report summaries to external stakeholders Public reporting Internal documents and data management Establishing management information systems to use evaluation data 26 2011 IREX CSP, Cindy Banyai, refocusinstitute.com Slide 28 7 - Give feedback Use results to improve management of ongoing policies/programs/projects Feed results into planning new policies/programs/projects Add reports to systems to improve organizational learning and accountability 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 27 Slide 29 Thanks for your attention! Please send questions & comments to photoevaluation@refocusinstitute. com cindy.banyai@gmail.com Slide 30 References AusGUIDElines (2003). The Logical Framework Approach. Canberra: Australian Agency for International Development. Banyai, Cindy. 2010. Community capacity and governance New approaches to development and evaluation. PhD diss., Ritsumeikan Asia Pacific University. International Fund for Agricultural Development (2010). Rural Poverty Report 2011: New realities, new challenges: new opportunities for tomorrows generation. Rome International Fund for Agricultural Development. Japan International Cooperation Agency - Office of Evaluation and Post Project Monitoring, Planning and Evaluation Department (2004). JICA Evaluation Handbook: Practical Methods for Evaluation. Tokyo: Japan International Cooperation Agency. Miyoshi, Koichi. (2008). What is Evaluation?. In Miyoshi, Koichi (Ed). Hyoka-ron wo Manabu Hito no tameni (For People Learning Evaluation Theory). (pp. 1- 16). Tokyo. Sekaishisosha. Patton, Michael Quinn (2002). Qualitative Research & Evaluation Methods. Thousand Oaks: Sage Publications, Inc. Weiss, Carol H. (1998). Evaluation. Upper Saddle River, NJ, USA: Prentice- Hall, Inc. 2011 IREX CSP, Cindy Banyai, refocusinstitute.com 29