Indicators Regional Workshop on the Monitoring and Evaluation of HIV/AIDS Programs February 14 –...
-
Upload
deirdre-allison -
Category
Documents
-
view
214 -
download
0
Transcript of Indicators Regional Workshop on the Monitoring and Evaluation of HIV/AIDS Programs February 14 –...
Indicators
Regional Workshop on the Monitoring and Evaluation of HIV/AIDS Programs
February 14 – 24, 2011New Delhi, India
Session Objectives Understand how indicators are linked to
frameworks Describe how to operationalize indicators Identify the role of indicators at different levels
(national, sub-national, project) and the linkages between them
List sources of indicators that are international standards
Select indicators and define indicators for an M&E plan
An Indicator is…
a variable (its value changes)
that measures (objective calculation of value)
key elements of a program or project Inputs, processes, outputs, outcomes
Indicators provide critical M&E data at every level (and stage) of program implementation
Inputs, Process Was the program carried out as planned?
How well was it carried out?
Outputs, Results Did the expected change occur?
How much change occurred?
Outcome, Impact Has the outcome changed in desired direction?
Does the change signal program “success”?
INPUT
• Human and financial resources
• Development of training materials
PROCESS
• Conduct one PMTCT training workshop in each district for providers
OUTPUT
• Providers trained in updated PMTCT service provision
OUTCOME
•Increased use of PMTCT services
IMPACT
• Reduced perinatal transmission of HIV
Logic Model IndicatorsLogic Model Indicators
Indicator: # of providers who have completed clinical training
Indicator: % of pregnant women who are HIV tested
Indicator: percent of HIV+ women receiving a complete course of ARV prophylaxis
Indicator: percent of infants HIV+ born to HIV+ women
Results Framework Example – PEPFAR funded ART Program
SO: Increased utilization of ART services
IR-1: Availability of quality services
IR-2: Demand for services
IR-1.1: Increase ART sites
IR-1.2: Supply sites with ARVs
IR-1.3: Training for providers
IR-2.1: Increase knowledge ofART
IR-2.2: Increase referral from VCT
Results Framework Indicators PEPFAR funded ART Program
IR-1: Availability of quality services
IR-1.1: Increase ART sites
IR-1.2: Supply sites with ARTs
IR-1.3: Training for providers
# of districts with at least one facility providing ART services in line with national standards
# of ART sites
% of ART delivery points experiencing stock-outs in the preceding six months
# of health workers trained on ART delivery in accordance with national or international standards
Results Framework Indicators PEPFAR funded ART Program
SO: Increased utilization of ART services
# of HIV + persons receiving ART therapy
Indicator Pyramid
District, Facility, ProjectIdentify progress, problems, and challenges
National/Sub-national Assess effectiveness of response
Reflect goals/objectives of national/sub-national response
Number of Indicators
Decreases
Increases
GlobalCompare countries
Overview world-wide situation
Linkages between levels
Often, indicators at the higher levels in the pyramid are linked to those at the lower levels.
Data may be collected at the district level and passed up to the national level and on up to the global level.
Requires an M&E system to support data flow, compilation, and aggregation
Characteristics of good indicators
Valid: accurate measure of a behavior, practice or task
Reliable: consistently measurable in the same way by different observers
Comparable: can be measured in different contexts or time periods
Non-directional: subjective criteria not part of definition
Characteristics of good indicators (cont.)
Precise: operationally defined in clear terms Measurable: quantifiable using available tools and
methods Timely: provides a measurement at time intervals
relevant and appropriate in terms of program goals and activities
Programmatically important: linked to a public health impact or to achieving the objectives that are needed for impact
Common Indicator Metrics Counts
# of providers trained
# of condoms distributed
Calculations: percentages, rates, ratios
% of facilities with trained provider
% of FSW who used a condom at last sex with a client
Index, composite measures
Index on infection control prevention
DALY (Disability Adjusted Life Years)
Thresholds
Presence, absence
Pre-determined level or standard
Operationalizing indicators
Establish exactly how a given concept / behavior
will be measured precise definition and metric
how the value will be reliably calculated (anyone using the same data will arrive at
exactly the same indicator value)
Always specify the details!
who/what qualifies to be counted, and when
Number of providers trained in PMTCT services
“providers”: any clinician providing direct clinical services to women seeking ANC at government health facilities
“trained”: attended a two-week training course on PMTCT in past year
Always specify the details!
How to calculate it
Percent of health facilities with a provider trained in PMTCT services
“Numerator”: Number of public facilities with a provider who attended a two-week PMTCT training course in past year
“Denominator”: Number of public facilities offering ANC services in the past year
Always specify the details!Threshold indicator for VCT program
VCT centers with minimum conditions to provide quality services
• Facility provides quality VCT services if the following necessary structural elements are present– Trained staff– Adequate privacy for counseling– Systems for maintaining confidentiality– Directory of services for referral– Adequate conditions for ensuring quality control of specimen
tests
LimitationsAll indicators have limitations, even those
commonly used:
Blood safety: cannot monitor private facilities adequately Sexual behavior (e.g. condom use, number of partners):
self reporting bias Sero-surveillance: get biased population (pregnant
women and other populations) Population-based HIV prevalence: refusal bias ,
sampling bias
Frequency of Reporting on Indicators Input/Process: Continuously
Output: Quarterly, semi-annually, or annually
Outcome: 1-3 years
Impact: 2-5 years
Managing indicator systemsBasic Indicator Matrix
Indicator
Data Source Frequency
Decision Points/Comments
Outputs# of peer outreach contacts in the past 12 month
Program records
Quarterly Disaggregate by district
Outcomes% of FSW who used a condom at last sex with a client
Target group survey
1-3 years Disaggregate by age and type (brothel, non-brothel based)
The Role of Target Setting
Important element of strategic planning
Tracking progress towards achieving targets assists with resource allocation and improves program management
Often a requirement for performance-based funding
Main Steps in Target Setting
Define the populations and subpopulations in need of services (treatment, diagnostic, care, prevention, etc)
Assess existing coverage to identify gaps
Determine potential achievements within the time frame given resource availability, opportunities, and constraints
⇒ Set ambitious but realistic targets!
Indicator Reference Sheets
Assist in detailed documentation of indicators
Clear definition of indicator, numerator, and denominator
Collection details – data source, frequency, who responsible
Content of Indicator reference sheets (see handout)
The linked result
Precise definitions for every term used
Reason for selection, how to interpret
Data requirements, data quality issues
How to measure and calculate
Data source, responsibilities, and frequency
Issues, limitations, significance
Data table (baseline and target values)
Guiding Principles to selecting indicators
Ensure that the indicators are linked to the program goals and are able to measure change
Ensure that standard indicators are used to the extent possible
Consider the cost and feasibility of data collection and analysis.
Keep the number of indicators to the minimum and include only those needed for program and management decisions or for reporting.
Good indicators: A Synopsis
Provide information useful for program decision-making Are consistent with international standards and other
reporting requirements, as appropriate Are defined in clear and unambiguous terms
Non-directional, “independent,” and SMART Have values that are:
Easy to interpret and explain Precise, valid and reliable measures Comparable across relevant population groups,
geography, other program factors, as needed
Indicator Guides National AIDS Programmes. A guide to monitoring and evaluation.
UNAIDS (2000)
National AIDS Programs. A guide to indicators for monitoring and evaluating national HIV/AIDS prevention programs for young people. WHO (2004).
National guide to monitoring and evaluating programmes for the prevention of HIV in infants and young children. UNAIDS/WHO (2004)
National AIDS Programs. A guide to monitoring and evaluating HIV/AIDS care and support. WHO (2004).
National AIDS Programmes. A guide to indicators for monitoring and evaluating national antiretroviral programmes. WHO (2005).
Indicator Guides (cont.) Guide to monitoring and evaluation of the national response for
children orphaned and made vulnerable by HIV/AIDS. UNAIDS/WHO (2005)
A framework for monitoring and evaluating HIV prevention programs for most-at-risk populations (UNAIDS 2007)
Monitoring and Evaluation Toolkit. HIV/AIDS, Tuberculosis, and Malaria. GFTAM (2009).
Monitoring the Declaration of Commitment on HIV/AIDS. Guidelines on construction of core indicators. UNAIDS (2009)
….and others!
Additional References
HIV/AIDS Survey Indicators Database
www.measuedhs.com/hivdata
UNAIDS 2010. Indicator Standards: Operational Guidelines for Selecting Indicators for the HIV Response.
http://www.globalhivmeinfo.org/
UNAIDS Indicator Registry
http://www.indicatorregistry.org/
Not everything that can be counted counts, and not
everything that counts can be counted.
Albert Einstein
Group Activity: Adding Indicators to a Framework
Assemble group-work members
Use the worksheet “Linking Indicators to Frameworks”
Take about one hour to complete sheet
Resulting framework with indicators will to presented back by a group representative (5 minutes for each group)
The selection of indicators will form the basis of your plan, You will have the opportunity to revise this further based on feedback
MEASURE Evaluation is a MEASURE project funded by the
U.S. Agency for International Development and implemented by
the Carolina Population Center at the University of North Carolina
at Chapel Hill in partnership with Futures Group International,
ICF Macro, John Snow, Inc., Management Sciences for Health,
and Tulane University. Views expressed in this presentation do not
necessarily reflect the views of USAID or the U.S. Government.
MEASURE Evaluation is the USAID Global Health Bureau's
primary vehicle for supporting improvements in monitoring and
evaluation in population, health and nutrition worldwide.