Evaluation of surveillance systems

Günter Pfaff 2009/10 / Viviane Bremer 2008 / Preben Aavitsland / FETP Canada

Günter Pfaff

17th EPIET Introductory CourseLazareto, Menorca, Spain

September – October, 2011

The surveillance loop

Health Care System Public Health Authority

Event Data

InformationIntervention(Feedback)

Reporting

Analysis & Interpretation

Decision

Importance of evaluation

Obligation Does the system deliver?

Credibility of public health service

In reality Often neglected

Basis for improvements

Learning process EPIET training objective

”Do not create one until you have evaluated one”

Detect trends? Epidemics?Provide estimates of morbidity and mortality? Identify risk factors?Stimulate epidemiologic research?Assess effects of control measuresLead to improved clinical practice?Lead to new/improved control measures?Lead to better advocacy and increased

funding?

Does the surveillance system…

Simplicity Flexibility Acceptability Data quality Sensitivity and Predictive

value positive (PvP) Capture-recapture

Representativeness Timeliness

Criteria to look at

CDC guidelines

SimplicityAs simple as possible while meeting the objectives Structure

Information needed Number and type of sources Training needs Number of information users

Functionality Data transmission System maintenance Data analysis Information dissemination

Components of system

• Population under surveillance

• Period of data collection

• Type of information collected

• Data source

• Data transfer

• Data management and storage

• Data analysis: how often, by whom, how

• Dissemination: how often, to whom, how

Confidentiality, security

Flowchart (HIV in Norway)

Referencelaboratory

Primary HIV reporting form,Blood sample for HIV test laboratory part 1

Lab report and HIV reporting form

HIV reporting form, part 2HIV infection Primary care (Prompting if necessary) National Institute

physician of Public Health

AIDS reporting formAIDS Hospital physician Semiannual check

Oral informationDeath, emigration Semiannual check

Patient

Flexibility

Ability of the system to accommodate changes

New event to follow-up New data about an event New sources of information

Acceptability

Willingness to participate in the system Participation (%) of sources Refusal (%) Completeness of report forms Timeliness of reporting

Acceptability Factors influencing the willingness to

participate Public health importance Recognition of individual contribution Responsiveness to comments/suggestions Time burden Legal requirements Legal restrictions

Data quality

Completeness• Proportion of

blank / unknown responses

• Simple counting

Validity• True data?

• Comparison Records inspection Patient interviews ...

Completeness of informationInformation

Total Total

records records

No. No. (%) No. No. (%)

Person

Name 703 703 (100) na

Birth date 703 703 (100) na

Birth month and year 703 703 (100) 1491 1489 (100)

Sex 703 703 (100) 1491 1491 (100)

Municipality of residence at HIV-diagnosis 703 703 (100) 1491 1479 (99)

Country of birth 703 703 (100) 1491 1489 (100)

If not Norway

Reason for stay in Norway 109 100 (92) 592 551 (93)

Length of stay in Norway at HIV-diagnosis 109 62 (57) 592 352 (59)

Place

Infection acquired in Norway or abroad 703 334 (48) 1491 998 (67)

Cases acquired abroad

Country where infection was acquired 196 171 (87) 665 606 (91)

AIDS cases HIV cases without AIDS

Records with

item filled in

Records with

item filled in

Exposed

Clinical specimen

Symptoms

Pos. specimen

Infected

Seek medical attention

Report

Sensitivity

= reported true cases total true cases

= proportion of true cases detected

Disease

Notified

+

+ -

Total sick Total not sick

Total not notified

Totalnotified

True -

False +

False -

True +

-

Sensitivity = True + / Total sick

Specificity = True - / Total Not sick

PVP = True+ / Total notified

Sensitivity

-

-

Sensitivity versus specificity

The tiered system: confirmed, probable, possible

Frequent "false-positive" reports Inappropriate follow-up of non-cases Incorrect identification of epidemics

Wastage of resourcesInappropriate public concern (credibility)

Consequences of low PvP

Measuring sensitivity

• Find total true cases from other data

sources medical records

disease registers

special studies

• Capture-recapture study

Capture-recapture

• Used for counting total number of individuals in population using two or more incomplete lists

• Originally used in wildlife counting(birds, polar bears, wild salmon…)

Uses in epidemiology

• Estimate prevalence or incidence from incomplete sources

• Evaluate completeness of a surveillance system

Principles

• Two/more sources of cases with disease Lists, registries, observations, samples

• Estimate total number in the source population (captured and uncaptured) from the numbers of captured in each capture

Assumptions

1. The population is closed No change during the investigation

2. Individuals captured on both occasions can be matched No loss of tags

3. For each sample, each individual has the same chance of being included Same catchability

4. Capture in the second sample is independent of capture in the first The two samples are independent, pYZ = pY pZ

Seaworld Oberhausen, August 2010

Daddy, how many fish are in the aquarium?

Your options as a scientist

• Don‘t answer => Expect repeat question

• Answer something => „How do you know?“

• Consult an expert

• Estimate yourself

Meet the expert - „Pulpo Paul“• Has nine brains and three hearts

• Managed to predict all German games during the 2010 Football World Cup right

• Predicted accurately the finale Netherlands-Spain

Binomial distributions

only

http://www.elpais.com/articulo/gente/tv/Muere/pulpo/Paul/elpepugen/20101026elpepuage_4/Tes

Two-source model

Source Z

Source Y

b a c

x=?

N=?

N= a + b + c + x

Z1

Y1

Two-source analysis

Yes No

Yes a b Z1 = a + b

No c x

Y1 = a + c N = a + b + c + x

Source Y

Source Z

N = Y1 Z1 / a

Sensitivity of Y Ysn = Y/N = (a+c)/NSensitivity of Z Zsn = Z/N = (a+b)/N

How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“

How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, Dinner on Monday, 10 October 2011 – Case definiton: „Countable heads“, n=33

3

4 4

4

5

4

2

3

3

1

Hand does not meet our case

definition

This is our first view

How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=18

3

3

4

2

6

This is our second view

How many participants at the course?

• Capture: Source ”View #1”• Recapture: Source ”View #2”

• Estimations

• Assumptions hold?

Number of participants

N = 33 * 18 / 13 = 47

Sensitivity of View # 1 Sn1 = 33/47 = 70.2%Sensitivity of View # 2 Sn2 = 18/47 = 38.3%

Yes No

Yes 13 20 View #1 = 33

No 5 x

View # 2 = 18N = 13 + 20 + 5 + x

Source View #2 – After Dinner Tutorial

Source View #1Dinner

How many persons are in the EPIET 2011 Introductory Course?Isla del Lazareto, After Dinner Tutorial on Monday, 11 October 2011 – Case definition: “Countable heads“, N=20

3

3

4

2

6

This is our second view (revisited)

+ 2

Number of participants

N = 33 * 20 / 13 = 51

Sensitivity of View # 1 Sn1 = 33/51 = 64.7%Sensitivity of View # 2 Sn2 = 20/51 = 39.2%

Yes No

Yes 13 20 View #1 = 33

No 7 x

View # 2 = 20N = 13 + 20 + 7 + x

Source View #2, revised – After Dinner Tutorial

Source View #1Dinner

So, just how many are there?

2

9 25

Isla del Lazareto, Katharina‘s Lecture, Monday, 11 October 2010 – Case definition: “Persons in room“, N=53

9

18

9

30

5 off screen

The problem with the X:Finding a comprehensive view

Assumptions may not hold

1. The population is closed - Usually possible

2. Individuals captured on both occasions can be matched - OK if good recording systems

3. For each sample, each individual has the same chance of being included - Rarely true

4. Capture in the second sample is independent of capture in the first - Rarely true

Sources are independent(most important condition)

Being in one source does not influence the probability of being in the other source

bc

ad OR

OR > 1 (positive dependence): underestimates N

OR < 1 (negative dependence): overestimates N

Yes No

Yes a b Z1

No c d

Y1 N

Source Y

Source Z

Dependent sources

• Estimation of number of IVDU in Bangkok in 1991 (Maestro 1994)

• Two sources used: Methadone programme (April – May 1991) Police arrests (June – September 1991)

• Methadone Need for drugs Probability of being arrested = negative dependence, overestimation of N

Usefulness of capture-recapture

• If conditions are met Great potential to estimate population size by

using incomplete sources Cheaper than exhaustive registers or full counting

• Two sources Impossible to quantify extent of dependence

• Multiple sources Can adjust for dependence and variable

catchability

Examples of capture-recapture

• STDs in The NL Reintjes et al. Epidemiol Infect 1999

• Foodborne outbreaks in France Gallay et al. Am J Epidemiol 2000

• Pertussis in England Crowcroft et al. Arch Dis Child 2002

• Invasive meningococcal disease Schrauder et al. Epidemiol Infect 2006

Representativeness A representative system accurately describes

Occurrence of a health event over time Distribution in the population by place and time

Difficult to determine Compare reported events with actual events Characteristics of the population Natural history of condition, medical practices Multiple data sources

Related to data quality, bias of data collection, completeness of reporting

Timeliness

Disease onset

Diseasediagnosed

Reporting

of event

Action taken

Analysis and interpretation

Clinician, labs Public Health Authorities

SensitivityRepresentativeness

Predictive value positive

TimelinessAcceptability

FlexibilitySimplicity

Cost

Buehler’s balance of attributes

• Recommendations of evaluation Continue Revise Stop

If revising Increase participation rate of sources Simplify notification Increase the frequency of feedback Broaden the net . . . Activate data collection

Improving surveillance systems

Surveillance is like archeology of the immediate past –It requires your responsible imagination of an invisible reality.

Carnunthum, Austria

Corollary

Thank you!

Literature

• CDC. Updated guidelines for evaluating public health

surveillance systems. MMWR 2001; 50 (RR-13): 1-35

• WHO. Protocol for the evaluation of epidemiological

surveillance systems. WHO/EMC/DIS/97.2.

• Romaguera RA, German RR, Klaucke DN. Evaluating

public health surveillance. In: Teutsch SM, Churchill RE,

eds. Principles and practice of public health surveillance,

2nd ed. New York: Oxford University Press, 2000.

Reading on capture-recapture

• Wittes JT, Colton T and Sidel VW. Capture-recapture models for assessing the completeness of case ascertainment using multiple information sources. J Chronic Dis 1974;27:25-36.

• Hook EB, Regal RR. Capture-recapture methods in epidemiology. Methods and limitations. Epidemiol Rev 1995; 17: 243-264

• International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation I: History and theoretical development. Am J Epidemiol 1995;142:1047-58

• International Working Group for Disease Monitoring and Forecasting. Capture-recapture and multiple-record systems estimation II: Applications in human diseases. Am J Epidemiol 1995;142:1059-68