Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work
-
Upload
rbfhealth -
Category
Healthcare
-
view
203 -
download
1
Transcript of Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work
![Page 1: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/1.jpg)
Verification of Results
Group Work
Petra Vergeer | Senior Health Specialist
![Page 2: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/2.jpg)
Overview
1
• Operational data from RBF programs is of limited use
unless it is accurate.
• A cross-case analysis was carried out to expand knowledge
about verification processes and practices. o Afghanistan, Argentina, Burundi, and Rwanda
• Introduction of verification “pathway tool” o Initial consultation to obtain feedback to further develop tool
![Page 3: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/3.jpg)
Verification Pathway Tool
![Page 4: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/4.jpg)
Examine quantity verification error rates
1
Facility-level patterns
Indicator-level patterns
Generalized
across all
contracted
parties
Localized to specific
facilities (by
geographic area)
Localized to
specific types
of facilities
Indicators with
complex
compliance
criteria
Indicators that are
rewarded more
frequently and/or
have higher patient
volume
Indicators
rewarded at a
higher level
Examine quantity
verification error rates
Activities to
explore
Activities to
explore
![Page 5: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/5.jpg)
Examine patient tracing error rates
Examine patient tracing error
rates
Facility-level patterns
Indicator-level patterns
Generalized
across all
contracted
parties
Localized to specific
facilities (by
geographic area)
Localized to
specific types
of facilities
Indicators with
complex
compliance
criteria
Indicators that are
rewarded more
frequently and/or
have higher patient
volume
Indicators
rewarded at a
higher level
Activities to
explore
Activities to
explore
![Page 6: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/6.jpg)
Examine counter-verification error rates Verifier-level patterns
Indicator-level
patterns
Generalized
across all
verified
parties
Localized
to a
specific
verifier
Localized
to a
specific
type of
facility Examine quantity
and quality counter-verification error
rates (Assume data are at level
of contracted party.)
Indicators with
complex
compliance
criteria
Indicators that are
rewarded more
frequently and/or
have higher patient
volume
Indicators
rewarded at a
higher level
Activities to
explore
Activities to
explore
![Page 7: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/7.jpg)
Factors influencing verification: a
conceptual framework Context
Verification Characteristics Impact on accuracy,
cost, sustainability
RBF Characteristics RATIONALE FOR RBF CONTRACT TYPE
USE OF RBF RESULTS
Improving health outcomes/HSS Relational
Payment, improving performance
Financial accountability/Cost control Classic
Transparency, Naming and Shaming
Monthly Annual
Yes
Large
Whole universe Risk-based approach Internal
Verification Results and Their Use
FREQUENCY
ALLOWABLE ERROR MARGIN
SAMPLE SIZE INSTITUTIONAL SETUP ADVANCE WARNING No
Small
Third party
Learning, Error correction Cost recovery, Sanction
PAYMENT FREQUENCY Monthly Annual
POLITICAL ENVIRONMENT GOVERNANCE CULTURE
![Page 8: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/8.jpg)
Potential causes of generalized error rates:
verification • Knowledge and tools
o Insufficient training for providers, data entry clerks
o Data collection tools insufficiently clear
o Explore validity of verification methods
• Allowable error margin o Assess error margin
• Advance warning
• Institutional setup o There is a conflict of interest between the verifier and the contracted party
• Sample size o Data are “noisy”
• Frequency o Verification/counter-verification occurs too infrequently to deter gaming
• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting
Question: What are the causes in your country?
Back to
main slide
![Page 9: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/9.jpg)
Potential causes of generalized error rates:
patient tracing • Knowledge and tools
o Insufficient training for providers, data entry clerks
o Data collection tools insufficiently clear
o Explore validity of verification methods
• Allowable error margin o Assess error margin
• Advance warning
• Institutional setup o There is a conflict of interest between the verifier and the contracted party
• Sample size o Data are “noisy”
• Frequency o Verification/counter-verification occurs too infrequently to deter gaming
• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting
Question: What are the causes in your country?
Back to
main slide
![Page 10: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/10.jpg)
Potential causes of generalized error rates:
counter-verification • Knowledge and tools
o Insufficient training for providers, data entry clerks
o Data collection tools insufficiently clear
o Explore validity of verification methods
• Allowable error margin o Assess error margin
• Advance warning
• Institutional setup o There is a conflict of interest between the verifier and the contracted party
• Sample size o Data are “noisy”
• Frequency o Verification/counter-verification occurs too infrequently to deter gaming
• Verification results and use o Incentives and sanctions are not sufficient to mitigate mis-reporting
Question: What are the causes in your country?
Back to
main slide
![Page 11: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/11.jpg)
Potential causes of error rates localized to a
specific geographic area: verification
• Knowledge and tools o Insufficient support for contracted parties
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency o Verification occurs too infrequently to deter gaming
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 12: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/12.jpg)
Potential causes of error rates localized to a
specific geographic area: patient tracing
• Knowledge and tools o Insufficient support for contracted parties
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency o Verification occurs too infrequently to deter gaming
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 13: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/13.jpg)
Potential causes of error rates localized to a
specific verifier: counter-verification
• Knowledge and tools o Insufficient support for verifiers
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency o Verification occurs too infrequently to deter gaming
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 14: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/14.jpg)
Potential causes of error rates localized to a
specific type of facility: verification
• Knowledge and tools
• Allowable error margin o Explore whether error margin is appropriate
• Advance warning
• Institutional setup
• Sample size o Biased data due to insufficient sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 15: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/15.jpg)
Potential causes of error rates localized to a
specific type of facility: patient tracing
• Knowledge and tools
• Allowable error margin o Explore whether error margin is appropriate
• Advance warning
• Institutional setup
• Sample size o Biased data due to insufficient sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 16: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/16.jpg)
Potential causes of error rates localized to a
specific type of facility: counter-verification
• Knowledge and tools
• Allowable error margin o Explore whether error margin is appropriate
• Advance warning
• Institutional setup
• Sample size o Biased data due to insufficient sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 17: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/17.jpg)
Potential causes of high error rates among
indicators with complex compliance criteria:
verification
• Knowledge and tools o Confusing data collection tools
o Inadequate training for providers and data entry clerks
• Allowable error margin o Indicator too complex for accurate reporting
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 18: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/18.jpg)
Potential causes of high error rates among
indicators with complex compliance criteria:
patient tracing
• Knowledge and tools o Confusing data collection tools
o Inadequate training for providers and data entry clerks
• Allowable error margin o Indicator too complex for accurate reporting
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 19: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/19.jpg)
Potential causes of high error rates among
indicators with complex compliance criteria:
counter-verification
• Knowledge and tools o Confusing data collection tools
o Inadequate training for providers and data entry clerks
• Allowable error margin o Indicator too complex for accurate reporting
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 20: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/20.jpg)
Potential causes of high error rates among indicators that
are rewarded more frequently or have higher patient
volume: verification
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 21: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/21.jpg)
Potential causes of high error rates among indicators that
are rewarded more frequently or have higher patient
volume: patient tracing
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 22: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/22.jpg)
Potential causes of high error rates among indicators that
are rewarded more frequently or have higher patient
volume: counter-verification
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 23: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/23.jpg)
Potential causes of high error rates among
indicators that are rewarded at a higher level:
verification
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup o Incentive amount encourages gaming
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 24: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/24.jpg)
Potential causes of high error rates among
indicators that are rewarded at a higher level:
patient tracing
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup o Incentive amount encourages gaming
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 25: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/25.jpg)
Potential causes of high error rates among
indicators that are rewarded at a higher level:
counter-verification
• Knowledge and tools
• Allowable error margin
• Advance warning
• Institutional setup o Incentive amount encourages gaming
• Sample size
• Frequency
• Verification results and use
Question: What are the causes in your country?
Back to
main slide
![Page 26: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/26.jpg)
Examples of options to address high error rates
• Visit small number of facilities to investigate underlying
causes of error rate patterns
• Increase sanction amount/proportion
• Conduct additional trainings for providers, data entry clerks
• Provide additional support on an ongoing basis for
providers, data entry clerks
![Page 27: Annual Results and Impact Evaluation Workshop for RBF - Day One - Verification of Results Group Work](https://reader031.fdocuments.us/reader031/viewer/2022030117/589ee7b21a28abe97f8b4e7f/html5/thumbnails/27.jpg)
Thank You