Quality Management of Case Surveillance Systems

Session Overview We will:

Discuss what defines Quality when considering case surveillance

Articulate what value Quality brings to case surveillance

Consider how Quality can be assessed, assured, and improved upon

Quality Management of Case Surveillance Systems

Quality Management typically entails:

But wait… Why are we talking about Surveillance and M&E?

…. aren't they different things?

M&E vs. Surveillance

M&E Surveillance Aggregate counts Measure of criteria Measure vs. criteria Shows service/process

trends over time

Number of people tested Number of HIV+ tests Number of people on

ART

Unique, unduplicated counts Measure of disease burden Measure of influencing factors Shows disease-specific trends

over time

Number of unique people w/HIV Distribution of cases Factors influencing transmission

Number of unique people on ART Trends in ART access outcomes

But wait… Why are we talking about Surveillance and M&E?

…. aren't they different things?

Yes….. But…… How else will you know if your surveillance system gives you quality data?

M&E of Surveillance Measure of criteria Measure vs. criteria Shows service/process trends over time

Do we have a high-quality system? How is our surveillance system working? How can we improve the surveillance system?

Do we have high-quality data? Are the data reliable? Should we trust our data? How can we improve our data?

What is Quality, with respect to Case Surveillance?

Degree of Excellence of …What?

What is Quality, with respect to Case Surveillance?

Degree of Excellence of …What?

Yes!

Why Monitor and Evaluate a Case Surveillance System?

Success +

Change

Quality+

Confidence

Why is Quality Important for Case Surveillance?

Being able to speak to the quality of Collection- the data collected Collation- the process used to collect,

clean and manage the data Analysis- the trends, and findings

presented Interpretation- the action items

suggested Dissemination- the data given to key

partners

Will increase your confidence in the data and the process, and that will then better guide how others will use and react to the data.

Accuracy of data

Accuracy of

assumptions

Accuracy of action

taken

When should a quality management process be implemented?

Quality Management is a continuous process It should be planned early on It should begin the moment a program is initiated It should be routine and ongoing Its findings should be integrated into an adapting

and evolving program

How to Monitor and Evaluate a Case Surveillance System?

How to Monitor and Evaluate a Case Surveillance System

A. Define criteria of interest What is it that you want to know that will

instill confidence in your system and your data? Are all expected cases reported? Have all expected sites reported? Are all (key) fields entered on the case report

form? Are all fields entered correctly? Are the data accurate? Is reporting timely?

A. Define criteria of interestSample quality criteria for case surveillance

Influence Data Quality

Influenced by Process

A. Define criteria of interest Related to the Process

Process Monitoring will allow you to understand: What is working well What are the gaps

in the

Data collection process Data reporting process Data cleaning process Data utilization process

Success or limitation in this area will influence data quality

A. Define criteria of interest Related to the Data

This will allow you to understand: What is working well What are the gaps

in the

Data collection process Data reporting process Data cleaning process

Support and technical assistance in this area will allow you to have stronger confidence in using the data

B. Set measures by which to assess How can you measure and perceive change in your

defined criteria? What are your objectives/standards?

B. Set measures by which to assess ProcessProcess Monitoring should be implemented in a routine manner

Is typically observational Site level Intermediate level National level

Criteria/measures will be defined by the Standard Operating Procedures (SOP)

What is the gold standard for data collection and transmission?

What is actually happening?

B. Set measures by which to assess Process

How are patient level data gathered? Self report? Proxy? Any verification? Ideally, patient-level data should be:

Self reported (not assumed or by proxy) Validated (with identification, with questions for

clarification) Inaccurate data, pseudonyms render case matching difficult.

How are case data transcribed? How many people touch each case? How many times

are data copied? Any double entry or validation? Ideally, data should be:

Copied or transcribed minimally (source is best!) Validated if copied or transcribed

Errors of transcription create inaccurate data.

B. Set measures by which to assess Process

How are cases reported? Is responsibility clear? Are timelines clear? Is there

supervision? Any validation? Ideally, there should be:

Clearly defined roles and responsibilities Clear timelines Routine supervision that includes validation

If cases are not reported in a routine and timely way, data will be incomplete.

How are data managed? Are data cleaned? Are cases matched or de-duplicated? At

what level does this occur? How many people are responsible for this?

Ideally, data should be regularly: Cleaned and deduplicate by few, but consistent people Followed-up upon if there are questions

If cleaning is not routine, and feedback given, data will continue to be poor.

B. Set measures by which to assessData - Representativeness Representativeness describes your ability to

accurately describe the disease over time and by population and place Are all expected cases a part of the national data

set? If 100 HIV+ tests were performed at a site, 100 HIV

case report forms should be completed and submitted

Are all sites, types of service, regions represented? If there are 100 testing/treatment sites, are all 100

reporting cases on a regular basis? Examples:•90% of cases from the last quarter were reported •95% of eligible sites reported in the last quarter

B. Set measures by which to assessData - Representativeness

A) Calculating Representativeness of reported casesAre all cases that you expect to be in the system in the system?

Overall

By Region, Site, Network, etc.

# of reported cases (in time period)# of expected cases (in time period)

X 100

# of reported cases (in time period)# of expected cases (in time period)X 100

This could also be called system completeness

B. Set measures by which to assessData - Representativeness

B) Calculating Representativeness of sites reporting casesAre all sites that you expect to report to the system reporting?

Overall

By Region, Site, Network, etc.

# of sites reporting cases# of expected sites

X 100

# of sites reporting cases# of expected sites X 100

B. Set measures by which to assess Data - Completeness Completeness describes what proportion of an

expected count is ‘there’.

Are all expected data fields are complete? What proportion of fields on the case report form are

complete? What proportion of cases are reported with the (X) case-

defining variables complete?

Examples:•95% of reported cases have all case defining variables complete•95% of reported cases have Date of HIV Diagnosis complete

B. Set measures by which to assessData - Completeness

B. Set measures by which to assessData - Timeliness Timeliness is a measure of the speed between

steps in the process - time from diagnosis to reporting Are the cases submitted and processed within a

reasonable time? Are you presenting this year’s data, or five years

ago?

Example:•85% of cases are reported within three months of diagnosis

B. Set measures by which to assessData - Timeliness Can be measured two ways

A) Median time between diagnosis and receipt of case report at MOH (or other authority)

Calculated by subtracting the months between the date of report and the date of diagnosis.

Reporting delay values can be summed and the median calculated

B) Proportion of cases reported in a specified time period (some standards use 6 and 12 months )

Number of cases diagnosed within a year and reported within 6 months of diagnosis

Number of cases diagnosed and reported for that diagnosis year

Before measuring timeliness, determine representativeness of reportingIncomplete reporting will overestimate timeliness

B. Set measures by which to assessData - Accuracy Accuracy describes the degree to which your data

are ‘true’. Your measure of accuracy will help you establish a degree

of confidence (or not!) in your data

Data Accuracy is influenced by: Collection of data from patient to forms

Does the patient tell you the correct data? Does the counselor record the correct data with correct

spelling? Does the counselor ask all questions (rather than guessing)?

To assess the accuracy of data reported or collected from the patient, you will need to use process/observational methods

B. Set measures by which to assessData - Validity Validity describes the degree to which your data

are ‘true’. Your measure of validity will help you establish a degree of

confidence (or not!) in your data Are males pregnant? Are people 120 years old? Were people diagnosed with HIV in 1958? Do people have negative CD4 values?

Initial data checks can be done on data sets to look for anomalies

Data Validity is influenced by the: Transcription of data from one form to another

Spelling, number sequence, correct code/box Paper to paper and/or paper to computerExample:•95% of reported cases have a valid Date of Birth reported

B. Set measures by which to assessData - Validity

C. Gather Data Data collection should be regular, routine, and ‘low

barrier’ Create templates and checklists Train and support site-level staff to do routine

monitoring and reporting Use regional/national staff to do mentoring and

supportive supervision of the process, and periodic monitoring

Start soon and start small Pilot your M&E process and measures See if you are able to find the answers you are looking

for Modify as needed

C. Gather Data - Design a Monitoring and Evaluation Process Define the Details

How will the M&E Process be implemented?

This is your M&E Plan Brief description of the project and the evaluation

framework Detailed description of the indicators Data collection plan

Description of the data sources Description of data collection tools

Plan for how monitoring and evaluation will occur Dissemination and utilization plan for information

gathered Description of M&E plan update process

SOP

C. Gather Data Local Monitoring Methods

Observation/supervision

Data validation Selection of 5% of records Can use re-abstraction methods

Data double entry

Data triangulation Registers + Case Report + Patient Record + Lab

Record

C. Gather Data Regional/National Monitoring Methods

Site visits/observation/supervision

Assistance with data validation

Support for data triangulation National M&E Reports + Case Report Data + Other

Aggregate Counts

Feedback!!!

D. Summarize findings With respect to your defined criteria and

measures Use simple tools and outlines/templates to do this

Do this initially to document a baseline so you can measure/assess improvement

Do this frequently to ‘test your hypotheses’ (step E. Act on the Findings) and see if a change is worth it

Share your findings with all involved staff at all levels This can motivate them

This can validate their effort

D. Summarize findings Use Data Locally; Regionally

Share findings and empower a clinical site to:

Improve the quality of surveillance activities Quality and timeliness of data and reporting Quality of patient interactions to capture complete data

Analyze and use surveillance data to improve clinical outcomes Look at the clinical indicators collected; is this aligned

with standards of care?

Better understand their local epidemic Who needs to be reached? Do programs need to be altered to do so?

D. Summarize findings Use Data Regionally; Nationally

Share findings and empower national/regional programs to:

Propose and/or implement national program changes More/different investment in surveillance More/different investment in HIV program activities

Clinical outcomes Prevention for high-risk populations Referral programs

Propose and/or implement national policy changes Case reporting mandate Revised clinical guidelines Anti-discrimination policies

E. Act on the findingsExamples of Quality Improvement

Issue Improvement Result

Hand delivery of paper case report form from site to National level means six-month delay - Poor timeliness - Poor representativeness

Implementation of electronic interface (like:) - 3G cards - Excel-based - Scan and email

More timely and representative submission of case report data

Hand-written completion of case report form with low legibility and high spelling errors - Poor validity

Semi-automation of case report form (drop-down menus, standard spelling)

More accurate and valid submission of case report data; fewer data entry errors

Single case report from completed by 4+ staff: intake; counselor; nurse or MD; lab technician - Incomplete data - Poor validity

Single case report from used by one-to-two staff; multiple forms linked by unique ID

More complete data received (number of cases; volume of data)

E. Act on the findingsSimple Quality Improvement process

E. Act on the findings Step 1: Identify Issue/Area for Improvement

What is the issue that concerns you? Break it down into small components

Which components can be modified? Select one component and plan/implement a test of

change

Example1 2 3 4 5 6 7 8 9 10 11 12

Total sites MESI (M&E sites) % MESI (SE sites) % HIV site (HASS) % HIV+ (M&E MESI) HIV+ (HASS) % Unique (HASS) %ARTIBONITE 24 18 75% 12 50% 13 54% 4,323 3,298 76% 2,667 19%CENTRE 17 9 53% 4 24% 7 41% 1,900 1,365 72% 978 28%GRANDE ANSE 16 6 38% 6 38% 6 38% 943 957 101% 644 33%NIPPES 4 4 100% 3 75% 4 100% 996 1,050 105% 783 25%NORD 15 14 93% 14 93% 13 87% 3,635 3,675 101% 3,079 16%NORD EST 8 5 63% 5 63% 5 63% 1,009 945 94% 698 26%NORD OUEST 9 7 78% 5 56% 5 56% 1,352 1,383 102% 1,142 17%OUEST 85 82 96% 43 51% 42 49% 17,204 15,966 93% 11,104 30%SUD 25 20 80% 14 56% 14 56% 2,314 2,283 99% 1,909 16%SUD EST 9 8 89% 4 44% 4 44% 605 625 103% 589 6%(blank) 13 81 742 916% 671 10%

212 173 82% 110 52% 126 59% 34,362 32,289 94% 24,264 25%

Not all sites are reporting casesGrand Anse is the most challenged

E. Act on the findings Step 2: Measure performance

Define and Measure Who What Where When How

Set an objective for what you would like to see

Example1 2 3 4 5 6 7 8 9 10 11 12

Total sites MESI (M&E sites) % MESI (SE sites) % HIV site (HASS) % HIV+ (M&E MESI) HIV+ (HASS) % Unique (HASS) %ARTIBONITE 24 18 75% 12 50% 13 54% 4,323 3,298 76% 2,667 19%CENTRE 17 9 53% 4 24% 7 41% 1,900 1,365 72% 978 28%GRANDE ANSE 16 6 38% 6 38% 6 38% 943 957 101% 644 33%NIPPES 4 4 100% 3 75% 4 100% 996 1,050 105% 783 25%NORD 15 14 93% 14 93% 13 87% 3,635 3,675 101% 3,079 16%NORD EST 8 5 63% 5 63% 5 63% 1,009 945 94% 698 26%NORD OUEST 9 7 78% 5 56% 5 56% 1,352 1,383 102% 1,142 17%OUEST 85 82 96% 43 51% 42 49% 17,204 15,966 93% 11,104 30%SUD 25 20 80% 14 56% 14 56% 2,314 2,283 99% 1,909 16%SUD EST 9 8 89% 4 44% 4 44% 605 625 103% 589 6%(blank) 13 81 742 916% 671 10%

212 173 82% 110 52% 126 59% 34,362 32,289 94% 24,264 25%

Only 38% of sites in Grand Anse have reported an expected HIV case to HASS in the last year.

Aim for 50% in one month; 75% in six months

E. Act on the findings Step 3: Brainstorm Cause of Issue

Gather information Go meet with local staff Understand the issue

Collectively come up with strategies for improvement Enable the local staff to come up with solutions Before you act… Consider what is actually in your

control. It is likely more effective to focus on those!

Example Issues

High turnover of nurses

Interruptions in supply of test kits (tests not done)

Case reports out of stock

Power outages

Staff have additional duties

Data collection incomplete

Unclear who is responsible

Possible Solutions Cross-train staff

Increase staff mentorship

Define/assign roles and responsibilities

Create inventory checklist, implement use, and communication of needs

Procure battery back-up

Plan, Do, Study, Act Circle

PLAN

DO

STUDY

ACT

E. Act on the findings Step 4: Generate and Plan Improvement Ideas Select ONE area and ONE element

Brainstorm: What is one thing we can do on one day to see

improvement? Who do we need to train/support to make this one

change? How will we measure if this creates an improvement?

Hints: Start small Start local Start quickly

E. Act on the findings Step 5: Implement Change

For the ONE area and ONE element

Support Implementation: Take a baseline measure Train/orient the key staff Implement the change for one hour/one day Measure the result

Hints: Start quickly Don’t over think it

E. Act on the findings Step 6: Study Results of Change

For the ONE area and ONE element

Evaluate the result What happened from baseline to post-evaluation? Did you see a change in the direction that you

wanted? What are some possible influencing factors?

Hints: Don’t over think it Use the data There is no failure… you can learn from all outcomes

E. Act on the findings Step 7: Act Accordingly

One – Expected improvement outcome Expand pilot Communicate Incorporate into Standard Operating Procedure

Two – Not quite expected outcome Refine idea Run through PDSA again

Three – No improvement Okay Refine theory Or move onto another process

F. Repeat process frequently Monitoring, evaluation and quality

improvement should be: Routine and ongoing Low-barrier Simple Implemented at multiple levels Implemented regularly by those close to the

source Implemented periodically by those further from

the source Designed to be inspiring, and not punitive

Final/Special Considerations

Staffing for Implementation M&E and quality improvement, should be routine

and well integrated into the surveillance and/or HIV program Consider having a point-person at the national level (25%-

50% FTE) to: Review data Produce reports Provide training and TA as needed

Consider having sub-national point-people (10%-50% FTE) to: Implement site visits (review and validation) Provide training and TA as needed

Consider having site-level point people (10% FTE) to: Implement data review and validation and ensure reporting, Request training and TA as needed

Methods of Implementation As you identify staff that can be involved in the

M&E and quality improvement process, consider: Empowering them Getting them excited to be a part of a positive

initiative Train them… in content and as trainers Encourage them to be SUPPORTIVE and

EMPOWERING, not punitive Enable them to engage in:

Routine and supportive supervisory initiatives Hands-on technical assistance to support improvement Being a part of the solution, not just directing it

Summary Quality is something important to measure in relation to

the process, inputs, and uses of case surveillance system and data.

Quality should be assessed routinely via monitoring, and periodically via point-in-time evaluation.

There are simple criteria that you can use to define how to assess and measure quality – this can be quick and easy.

Quality should be considered at all levels of the system: site level, sub-national level, and national level.

Findings from a quality assessment can (and SHOULD!) be used to improve quality.

Being able to speak to the quality of your data will improve your confidence in your system, data, analysis, and suggestions.

Thank You

Working Together to Plan, Implement, and Use

HIV Surveillance Systems