Jhpiego M&E Standards Process...

Jhpiego M&E Standards Process Guide Monitoring, Evaluation and Research Unit

© Jhpiego Corporation, 2013. All rights reserved. Suggested citation: Linda Fogarty, Eva Bazant, Anushree Ray, and Maya Tholandi (2013). Jhpiego M&E Standards Process Guide. Jhpiego: Baltimore, MD. Jhpiego’s Monitoring, Evaluation and Research Unit (MER) developed the M&E Standards in 2012 and this accompanying Process Guide in 2013. We are grateful to Megan Wysong, Reena Sethi, and Christina Maly, who provided critical input to the Process Guide. The developer of the accompanying scoring tool in Excel is Naresh Joshi of Jhpiego/India. Thanks go to Humberto Muquingue, Marya Plotkin, Webby Kanjipite, and Kenanao Motlhoiwa, who reviewed this guide and offered critical input. We would also like to acknowledge Jhpiego’s Publications unit for editorial and desktop publishing support.

Jhpiego M&E Standards Process Guide iii

Table of Contents

Introduction, purpose, and target audience of the M&E Standards Toolkit .......................... 1

What tools are available in the M&E Standards Toolkit? ........................................................ 2

How do we implement the M&E Standards? ............................................................................ 2

What are the nine M&E Standards? .......................................................................................... 5

How should we score each of the M&E Standards? ............................................................... 6

How else can we visualize the results of the M&E Standards Exercise? .............................. 8

What are the outputs of the M&E Standards Exercise and how can the results be shared? ........................................................................................................................................ 9

Appendicies

Appendix 1: Tips on Understanding Specific Standards or Verification Criteria ............... 11

Appendix 2: M&E Standards Tool: Scored Country Example .............................................. 13

Appendix 3: Additional Resources ......................................................................................... 18

iv Jhpiego M&E Standards Process Guide

Jhpiego M&E Standards Process Guide 1

Introduction, purpose, and target audience of the M&E Standards Toolkit

The purpose of this Process Guide and the companion M&E Standards Tool is to articulate a process that can be used by an organization’s offices to implement the standards. The main objectives are as follows: Ensure the quality of monitoring and evaluation (M&E) practices and, in turn, organizational

programs and results

Provide standardized guidance in a user-friendly format

Outline minimum requirements for sound M&E systems and processes to support an organization’s programs

These standards are to guide M&E activities in each of an organization’s offices and programs, including field offices in other countries for those organizations with an international portfolio. The main users are envisioned to be M&E and other staff, and in some cases technical advisors. Senior management should be involved in planning and debriefing meetings. As an organization grows, and the number and type of programs it implements globally also grow, a standard approach to M&E will help ensure the quality of M&E practices and, in turn, the organization’s programs and results. Quality M&E will allow us to tell our story with the best possible evidence. In a user-friendly format, the Jhpiego M&E Standards Tool articulates minimum requirements for M&E systems and processes for all of an organization’s programs. The standards are designed to be used by the organization’s offices and staff, domestically and internationally, to identify gaps in M&E and to develop plans to strengthen program and country M&E. This Process Guide is a companion document to the M&E Standards Tool. The Jhpiego M&E Standards were adapted from UNAIDS and the Monitoring and Evaluation Reference Group (MERG) 2008 document, Organizing Framework for a Functional National HIV Monitoring and Evaluation System. Jhpiego’s M&E staff at headquarters and in the field discussed the standards and selected nine standards with verification criteria. Similar to Jhpiego’s Standards-Based Management and Recognition (SBM-R®) process, adherence to the M&E Standards is measured using verification criteria. Each of the M&E Standards has between two and seven verification criteria (for more information on SBM-R, please see Jhpiego’s Standards-Based Management and Recognition: A Field Guide (Necochea and Bossemeyer 2007). The M&E assessment team reviews the description of the possible verification criteria scores, and chooses the score that fits best. The M&E Standards are to be used for each program, and eight of the standards are to be considered for the program. For organizations with international programs, one M&E Standard applies to the whole country office.

2 Jhpiego M&E Standards Process Guide

Figure 1. Jhpiego’s M&E Standards

What tools are available in the M&E Standards Toolkit?

There are five tools in the M&E Standards Toolkit:

Process Guide

M&E Standards Tool with nine standards (in Word or as pdf)

Action Plan template (in Excel or Word)

Scoring worksheet (in Excel)

Data visualization template (in PowerPoint) We recommend that the M&E Standards Tool be applied every year to identify gaps in order to strengthen M&E systems. How do we implement the M&E Standards?

The expectation is that the designated M&E lead will implement the M&E Standards Exercise annually. In large offices, the M&E lead will usually be the head of the M&E unit; in smaller programs, this may be the M&E Advisor or Officer. In very small programs or those with very constrained budgets, other staff may be responsible for program M&E, so they will lead and implement the exercise. The M&E lead, with the support of the senior management (including senior management personnel from the country office, where applicable), convenes the relevant staff (program and technical) and sets aside time (up to four meetings; meeting activities can be combined depending on staff availability and length of meetings). The purpose of each meeting and the steps of the process are outlined in Figure 2.


Figure 2. Process of Implementing M&E Standards

The meetings outlined below provide a general plan for applying the Standards Tool to the M&E programs managed by the organization. Meetings may be scheduled flexibly depending on the complexity of the country’s program portfolio and the availability of relevant staff. A. First meeting – INTRODUCTION (1–3 hours):

PURPOSE: Introduce the standards and the process to those who will implement the exercise, and create a plan for implementing the exercise.

SUGGESTED PARTICIPANTS: All M&E staff, and key program and technical staff.1

Introduce the standards and the process.

Plan the standards implementation for each program:2

− Determine who will attend each of the meetings and identify their roles.

− Plan meetings with other units, such as Finance, as needed.

− Remember that each program will be scored separately.

Answer questions about the process and the standards.

B. Second meeting – ASSESSMENT (2–3 hours per program):

PURPOSE: Complete the standards assessment for each program.

SUGGESTED PARTICIPANTS: All M&E staff, and key program and technical staff (program and technical staff can be invited when needed).

This should be done separately for each program in the office, such that each program has its own score.

Option 1: The team members score the programs individually before the meeting, and then scoring is discussed at this meeting.

1 The specific participants at each meeting will vary widely according to the program staffing structure and their availability. This guide presents suggested participant lists, but the ultimate list will be decided according to program needs. 2 For organizations with international programs, you will need to implement the standards separately for each program in your country portfolio. For example, one program may be funded by the U.S. Government, another by an international nongovernmental organization, and a third by a foundation. Each will be assessed separately.

Introduction to M&E

Standards (1–3 Hours)

Results & Action

Planning (1–2 hours)

Assessment of Each Program

(2–3 hours)

Annual Review

Presentation to Country Leadership

(1 hour)


Option 2: The team reads through each standard together, scoring each criterion for the office and for the program.

Encourage members of the review team to openly discuss each criterion and reconcile/resolve disagreements to assign a score that fairly reflects the current status. NOTE: The focus here should be on the discussion itself. This discussion is more important than the ultimate reconciled score.

C. Third meeting – RESULTS PRESENTATION and ACTION PLANNING (1–2 hours per program):

PURPOSE: Share assessment results and plan for filling identified gaps.

SUGGESTED PARTICIPANTS: All M&E staff, key program and technical staff, and key members of field offices, where applicable, including senior management personnel from the field.

Present the results to senior management (headquarters and field) and other staff.

Develop an initial Action Plan that includes the person responsible for carrying out the action for each gap identified.

Invite staff who can help interpret the results and plan and implement actions to bridge identified gaps.

Option 1: M&E staff develop an Action Plan but results are presented to a larger group with technical and program staff, and leadership. All can react to the Action Plan created by M&E staff.

Option 2. All relevant staff come together to develop the final Action Plan.

D. Fourth meeting – PRESENTATION TO COUNTRY LEADERSHIP (1 hour):

PURPOSE: Share the organization’s finalized commitment to the M&E Action Plan.

SUGGESTED PARTICIPANTS: Key representative(s) from M&E staff, senior management from headquarters and field offices, and other program leaders as appropriate.

Within two weeks of the third meeting, the Action Plan related to the M&E Standards should be finalized and presented to the senior management.

E. Preparation for future annual meetings:

PURPOSE: Continue the process of M&E strengthening.

SUGGESTED PARTICIPANTS: All M&E staff and all other staff members listed on the Action Plan.

Hold Quarterly Action Plan check-up meetings to monitor Action Plan progress.

Begin the following year’s assessment with a review of the Action Plan to determine what actions were taken to strengthen program and office M&E.

The first standard is applicable to the entire office, while the subsequent eight M&E Standards are applied to each program individually and should be assessed for each program. Decide in advance at the first meeting which programs will be reviewed against the M&E Standards.


In some cases, it may be preferable to have M&E program and technical staff jointly score the M&E Standards. Alternatively, each M&E staff member can score the standards individually, and then come together to compare notes and reconcile responses for the final scores. You may want to discuss with senior management (headquarters and/or field) the most appropriate process for your program. What are the nine M&E Standards?

The nine M&E Standards are given in the table below, along with their definitions. See the full Standards document for the specific criteria.

NINE M&E STANDARDS DEFINITIONS

1. M&E is integrated into programs. Routine M&E is part of each program and implementation approach, starting with a sufficient budget allocated for M&E. M&E, program, and technical staff work closely to ensure high-quality monitoring and continuous data use through program and technical refinement.

2. The program has adequate M&E human capacity.

Adequate skilled human capacity exists at all levels of the M&E system to complete all tasks defined in the program M&E work plan and is supported by human resources management practices.

3. Each program has a logic model, Performance Monitoring (or Management) Plan (PMP) and M&E work plan.

For each program, a logic model, PMP, and M&E work plan are developed, implemented, and updated annually.

4. Routine program monitoring systems are in place and maintained.

Routine data formats, collection, flow, quality control, and entry that avoids double-counting are in place and followed by each program component or sub-grantee.

5. The program uses databases to facilitate routine M&E data capture and analysis.

The organization’s centralized reporting system, training management system, and other databases are regularly updated, and the database reports are used for donor reporting and to guide program decision-making.

6. The program assesses and maintains high data quality.

Methods for updating and maintaining high-quality data are in place and used.

7. The program plans and implements rigorous evaluations.

Activity related to evaluation research is included in annual work plans and conducted accordingly.

8. The program uses and shares program data for decision-making.

The program has a data analysis and dissemination plan in place and routinely shares and uses data for decision-making.

9. The office has an M&E plan outlining standard operating procedures.

An M&E plan is developed and regularly updated to reflect the M&E needs across all programs. This includes human and material resources, data collection procedures, timelines, and roles and responsibilities for implementation of M&E systems to support all programs.


How should we score each of the M&E Standards?

The M&E Standards Tool is designed to be used as an internal needs assessment. Each standard has several verification criteria that can be assigned a score of 1, 2, or 3, and the meaning of each score is provided in the full standards document. Three is the highest score that can be assigned and 1 is the lowest. Only whole numbers can be used (e.g., 1.5 is not acceptable in the Excel scoring tool). If a response is not applicable, indicate “N/A” and write a clear justification in the comments section. Also provide an explanation in the comments section for any score of 2 or below. The M&E Standards Tool, which defines each standard and its verification criteria, is included in the Toolkit. Please note that the M&E Standards Exercise can be done on paper, electronically, or both. We recommend using the Excel scoring tool for entry of verification criteria scores, because it automatically aggregates scores, highlights the areas of the Action Plan that need to be completed, and automatically produces graphs of the results to use as a dashboard. Scores can be done on paper, but it would need to be transferred to Excel. The Excel scoring tool is also included in the M&E Standards Toolkit. The same Excel file can record scoring over several years. In the Excel scoring tool file, the first tab is the home screen (see screenshot below). There are four sections (tabs at the bottom) of this Excel scoring tool: Data Entry, Summary Tables, Action Plan, and Charts. Click on the tabs at the bottom of the screen to advance to each section.

Click here to advance to each section of the Excel scoring tool.


Below is a screenshot of the Data Entry section. Fill in the score using the dropdown menu shown below. Provide explanations as needed in the column to the right of the score. This is the only area of the spreadsheet where data from the scores and explanations should to be entered.

The next section is the Summary Tables section. This section is automatically filled in from the Data Entry section of the Excel scoring tool. Check this page to ensure that no errors were made during data entry. If you need to go back to the main menu or switch to another section, click on the back button on the top left corner of the screen as shown below.

The third section of the Excel scoring tool is the Action Plan. Verification criteria with scores of 2 or 1 will automatically populate the Action Plan table. Use this sheet to plan next steps to address performance criteria that need improvement.


The final section of this Excel scoring tool shows different ways to visualize the data from the M&E Standards Exercise. These charts are automatically populated from the Data Entry section of the Excel scoring tool. Three kinds of charts are created:

A summary of achievements chart (shown below),

A column/bar chart for each standard, and

A radar chart. For future years, continue to use the same Excel scoring tool to record progress of programs. This will allow you to see improvements in programs over time (trends).

How else can we visualize the results of the M&E Standards Exercise?

The M&E Standards “Pathway to Quality” (template) is another way of visually presenting the results of the M&E Standards. The red, yellow, and green colors are assigned to the scores of 1, 2, and 3 respectively (like a stop light, green is GO, or the highest score, and red is STOP, which needs further attention and action).


Below is an example: M&E Pathway to Quality

What are the outputs of the M&E Standards Exercise and how can the results be shared?

The following are the outputs of the M&E Standards Exercise:

Completed Action Plan template (in Excel or Word)

Completed Excel scoring tool worksheet (in Excel)

Graphs (radial and bar charts) serving as dashboards

Completed M&E Pathway to Quality: Country Office Example data visualization template (in Word)

Once all outputs of the M&E Standards Exercise have been completed and shared with leadership, including field office management, where applicable, all outputs should be shared with headquarters. These will be kept confidential and used for planning technical assistance and as a reference for each office or country to follow its own progress.


Appendix 1: Tips on Understanding Specific Standards or Verification Criteria

This section is about specific verification criteria and some terms that may need clarification. This is a living list and will be modified as more of the organization’s offices or country programs have experience with the M&E Standards Exercise. Standards 1–8 should be filled out for each individual program. However, for organizations with international programs, Standard 9 should be filled out only once for the entire country office. In some offices, M&E systems (e.g., databases—Standard 6) are used across programs. Answers to these standards would therefore be the same for each program reviewed in that country. 1. M&E staff are systematically integrated into programs at all levels. 1.2. Measuring “Program budgets allocate 5–10% of program funds to M&E activities and human resources”: Frequently, no clear line exists between some program and M&E activities, for example, when program staff provide supervision and collect data during the same site visit, or formative data collection intended for program development is used for baseline data monitoring. Further, financial expenditures are often not tracked by M&E activities. Therefore we suggest using the following three methods to serve as a proxy comparison for the actual comparison of money spent on all program M&E costs (including LOE + activities + materials/supplies/infrastructure costs) and overall program budgets:

1. Total of annual salaries for all program M&E staff divided by total annual salaries for all program staff

2. Total of annual salaries for all program M&E staff divided by annual program budget

3. Total number of M&E staff for the program divided by all program staff As we gain experience using these proxies, we hope to develop more rigorous methods for determining the percentage of budgets allocated to M&E. 1.3. Routine meetings between M&E and program staff are regularly held (at least quarterly) to share most recent data on key indicators, compare with expenditures, and inform ongoing programming. What do we mean by “compare with expenditures”? For example, if one of our key indicators is “people trained” and we are at the end of Quarter 2, if we still have 50% of our targeted numbers to train in the year, do we also still have 50% of our training budget? If not, what do we plan to do? With these questions we are comparing a key indicator with expenditures to inform ongoing programming. 3. Note: each program should be assessed separately. 4.5. Implementation documentation may be monitoring activities and documenting challenges and results in the quarterly/annual reports, or it may be a lessons-learned, stand-alone document.


4.6. Note that responsibility for developing or implementing e-health and m-health approaches does not always reside within the M&E unit. Therefore, to assess this verification criterion, those responsible for developing or implementing must be included in the standards assessment exercise. 4.7. Include time for headquarters review if the program is coordinated or funded by headquarters. 6.1. Note that this reference to data quality assessment/audit is related to the program’s current PMP indicators, and does NOT require a complete data quality assessment of all service data recorded at a facility. 7.1. Note that peer-reviewed papers written on routine monitoring data fall into the category of activities listed in this criterion.


Appendix 2: M&E Standards Tool: Scored Country Example*

“Data Entry” Excel Scoring Tool Sheet Example

*Please note that the Excel scoring tool sheets may look slightly different in the updated 2013 version.


“Summary” Excel Scoring Tool Sheet Example


“Action Plan” Excel Scoring Tool Sheet Example


“Charts” Excel Scoring Tool Sheet Example 1: Column Chart


“Charts” Excel Scoring Tool Sheet Example 2: Radar Chart


Appendix 3: Additional Resources

Additional sources for general performance management planning, including evaluation, are listed below. These resources are mainly from the MEASURE Evaluation project. Performance management planning ADS Chapter 203: Assessing & Learning. http://transition.usaid.gov/policy/ads/200/203.pdf. ADS 200 Supplement: The Performance Management Toolkit. http://transition.usaid.gov/policy/ads/200/200sbn.pdf. ADS Chapter 578: Information Quality Guidelines. http://transition.usaid.gov/policy/ads/500/578.pdf TIPS Number 6: Selecting Performance Indicators http://transition.usaid.gov/policy/evalweb/documents/TIPS-SelectingPerformanceIndicators.pdf TIPS Number 7: Preparing a Performance Management Plan. http://pdf.usaid.gov/pdf_docs/pnadw107.pdf TIPS Number 12: Data Quality Standards. http://transition.usaid.gov/policy/evalweb/documents/TIPS-DataQualityStandards.pdf TIPS Number 13: Building a Results Framework. http://pdf.usaid.gov/pdf_docs/pnadw113.pdf Evaluation: Learning from Experience. USAID Evaluation Policy. http://transition.usaid.gov/evaluation/USAIDEvaluationPolicy.pdf TIPS Number 19: Rigorous Impact Evaluation. http://pdf.usaid.gov/pdf_docs/pnadw119.pdf UNAIDS and the Monitoring and Evaluation Reference Group (MERG) (2008). Organizing Framework for a Functional National HIV Monitoring and Evaluation System. Available at: http://www.unaids.org/en/media/unaids/contentassets/documents/document/2010/20080430_JC1769_Organizing_Framework_Functional_v2_en.pdf

Jhpiego M&E Standards Process...

Documents

Transcript of Jhpiego M&E Standards Process...